2023-02-13 05:00:28,757 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599 2023-02-13 05:00:28,770 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.client.TestMobRestoreSnapshotFromClientSimple timeout: 13 mins 2023-02-13 05:00:28,787 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=3, rsPorts=, rsClass=null, numDataNodes=3, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-02-13 05:00:28,787 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/cluster_911aa46c-724d-9809-31e5-34f51d4314f6, deleteOnExit=true 2023-02-13 05:00:28,787 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-02-13 05:00:28,788 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/test.cache.data in system properties and HBase conf 2023-02-13 05:00:28,788 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/hadoop.tmp.dir in system properties and HBase conf 2023-02-13 05:00:28,789 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/hadoop.log.dir in system properties and HBase conf 2023-02-13 05:00:28,789 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/mapreduce.cluster.local.dir in system properties and HBase conf 2023-02-13 05:00:28,790 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-02-13 05:00:28,790 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-02-13 05:00:28,904 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-02-13 05:00:29,268 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-02-13 05:00:29,273 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-02-13 05:00:29,273 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-02-13 05:00:29,273 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-02-13 05:00:29,274 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 05:00:29,274 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-02-13 05:00:29,274 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-02-13 05:00:29,274 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-02-13 05:00:29,275 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 05:00:29,275 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-02-13 05:00:29,276 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/nfs.dump.dir in system properties and HBase conf 2023-02-13 05:00:29,276 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/java.io.tmpdir in system properties and HBase conf 2023-02-13 05:00:29,277 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/dfs.journalnode.edits.dir in system properties and HBase conf 2023-02-13 05:00:29,277 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-02-13 05:00:29,277 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-02-13 05:00:29,781 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 05:00:29,785 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 05:00:30,053 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-02-13 05:00:30,204 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-02-13 05:00:30,218 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 05:00:30,255 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 05:00:30,285 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/java.io.tmpdir/Jetty_localhost_46013_hdfs____zgulve/webapp 2023-02-13 05:00:30,435 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:46013 2023-02-13 05:00:30,447 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-02-13 05:00:30,447 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-02-13 05:00:30,829 WARN [Listener at localhost/36489] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 05:00:30,920 WARN [Listener at localhost/36489] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 05:00:30,937 WARN [Listener at localhost/36489] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 05:00:30,945 INFO [Listener at localhost/36489] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 05:00:30,949 INFO [Listener at localhost/36489] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/java.io.tmpdir/Jetty_localhost_33929_datanode____.cmxs8p/webapp 2023-02-13 05:00:31,050 INFO [Listener at localhost/36489] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33929 2023-02-13 05:00:31,367 WARN [Listener at localhost/41039] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 05:00:31,378 WARN [Listener at localhost/41039] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 05:00:31,384 WARN [Listener at localhost/41039] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 05:00:31,386 INFO [Listener at localhost/41039] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 05:00:31,391 INFO [Listener at localhost/41039] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/java.io.tmpdir/Jetty_localhost_42841_datanode____fco732/webapp 2023-02-13 05:00:31,486 INFO [Listener at localhost/41039] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42841 2023-02-13 05:00:31,494 WARN [Listener at localhost/45415] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 05:00:31,514 WARN [Listener at localhost/45415] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-02-13 05:00:31,516 WARN [Listener at localhost/45415] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-02-13 05:00:31,518 INFO [Listener at localhost/45415] log.Slf4jLog(67): jetty-6.1.26 2023-02-13 05:00:31,523 INFO [Listener at localhost/45415] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/java.io.tmpdir/Jetty_localhost_40223_datanode____vq3eji/webapp 2023-02-13 05:00:31,629 INFO [Listener at localhost/45415] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40223 2023-02-13 05:00:31,637 WARN [Listener at localhost/42165] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-02-13 05:00:31,839 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5686e4252e0dd8c7: Processing first storage report for DS-bb34b1c4-c7c8-49fb-bf39-0ff0ba335269 from datanode 0d06ca22-fbcc-472f-b304-b58ae62322d7 2023-02-13 05:00:31,841 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5686e4252e0dd8c7: from storage DS-bb34b1c4-c7c8-49fb-bf39-0ff0ba335269 node DatanodeRegistration(127.0.0.1:42015, datanodeUuid=0d06ca22-fbcc-472f-b304-b58ae62322d7, infoPort=33347, infoSecurePort=0, ipcPort=45415, storageInfo=lv=-57;cid=testClusterID;nsid=78588261;c=1676264429858), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-02-13 05:00:31,841 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x811feec654633491: Processing first storage report for DS-d680ab8e-5805-4c53-a666-ef4d67b97527 from datanode 9c79f770-6370-4a49-9b3f-555b35dac1f9 2023-02-13 05:00:31,841 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x811feec654633491: from storage DS-d680ab8e-5805-4c53-a666-ef4d67b97527 node DatanodeRegistration(127.0.0.1:35559, datanodeUuid=9c79f770-6370-4a49-9b3f-555b35dac1f9, infoPort=45043, infoSecurePort=0, ipcPort=41039, storageInfo=lv=-57;cid=testClusterID;nsid=78588261;c=1676264429858), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 05:00:31,841 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc13fb54f4d71293f: Processing first storage report for DS-efdf400e-6075-477f-9e34-d836ad5a615a from datanode e3004d4a-839e-4f08-ad25-249a5adccc94 2023-02-13 05:00:31,841 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc13fb54f4d71293f: from storage DS-efdf400e-6075-477f-9e34-d836ad5a615a node DatanodeRegistration(127.0.0.1:38213, datanodeUuid=e3004d4a-839e-4f08-ad25-249a5adccc94, infoPort=43145, infoSecurePort=0, ipcPort=42165, storageInfo=lv=-57;cid=testClusterID;nsid=78588261;c=1676264429858), blocks: 0, hasStaleStorage: true, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 05:00:31,841 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x5686e4252e0dd8c7: Processing first storage report for DS-b2949c02-d89b-4144-99ef-8cfc667d26be from datanode 0d06ca22-fbcc-472f-b304-b58ae62322d7 2023-02-13 05:00:31,841 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x5686e4252e0dd8c7: from storage DS-b2949c02-d89b-4144-99ef-8cfc667d26be node DatanodeRegistration(127.0.0.1:42015, datanodeUuid=0d06ca22-fbcc-472f-b304-b58ae62322d7, infoPort=33347, infoSecurePort=0, ipcPort=45415, storageInfo=lv=-57;cid=testClusterID;nsid=78588261;c=1676264429858), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 05:00:31,842 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x811feec654633491: Processing first storage report for DS-9052b549-3ffe-480f-be52-9dbda24b5c8e from datanode 9c79f770-6370-4a49-9b3f-555b35dac1f9 2023-02-13 05:00:31,842 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x811feec654633491: from storage DS-9052b549-3ffe-480f-be52-9dbda24b5c8e node DatanodeRegistration(127.0.0.1:35559, datanodeUuid=9c79f770-6370-4a49-9b3f-555b35dac1f9, infoPort=45043, infoSecurePort=0, ipcPort=41039, storageInfo=lv=-57;cid=testClusterID;nsid=78588261;c=1676264429858), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 05:00:31,842 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc13fb54f4d71293f: Processing first storage report for DS-e27d3a35-af4e-48d8-8e7c-811c582a61cc from datanode e3004d4a-839e-4f08-ad25-249a5adccc94 2023-02-13 05:00:31,842 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc13fb54f4d71293f: from storage DS-e27d3a35-af4e-48d8-8e7c-811c582a61cc node DatanodeRegistration(127.0.0.1:38213, datanodeUuid=e3004d4a-839e-4f08-ad25-249a5adccc94, infoPort=43145, infoSecurePort=0, ipcPort=42165, storageInfo=lv=-57;cid=testClusterID;nsid=78588261;c=1676264429858), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-02-13 05:00:32,016 DEBUG [Listener at localhost/42165] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599 2023-02-13 05:00:32,074 INFO [Listener at localhost/42165] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/cluster_911aa46c-724d-9809-31e5-34f51d4314f6/zookeeper_0, clientPort=63943, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/cluster_911aa46c-724d-9809-31e5-34f51d4314f6/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/cluster_911aa46c-724d-9809-31e5-34f51d4314f6/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-02-13 05:00:32,086 INFO [Listener at localhost/42165] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=63943 2023-02-13 05:00:32,093 INFO [Listener at localhost/42165] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 05:00:32,095 INFO [Listener at localhost/42165] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 05:00:32,754 INFO [Listener at localhost/42165] util.FSUtils(479): Created version file at hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a with version=8 2023-02-13 05:00:32,754 INFO [Listener at localhost/42165] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/hbase-staging 2023-02-13 05:00:33,057 INFO [Listener at localhost/42165] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-02-13 05:00:33,499 INFO [Listener at localhost/42165] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=18 2023-02-13 05:00:33,530 INFO [Listener at localhost/42165] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 05:00:33,530 INFO [Listener at localhost/42165] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 05:00:33,531 INFO [Listener at localhost/42165] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 05:00:33,531 INFO [Listener at localhost/42165] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 05:00:33,531 INFO [Listener at localhost/42165] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 05:00:33,669 INFO [Listener at localhost/42165] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 05:00:33,739 DEBUG [Listener at localhost/42165] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-02-13 05:00:33,831 INFO [Listener at localhost/42165] ipc.NettyRpcServer(120): Bind to /172.31.14.131:45905 2023-02-13 05:00:33,841 INFO [Listener at localhost/42165] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 05:00:33,843 INFO [Listener at localhost/42165] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 05:00:33,865 INFO [Listener at localhost/42165] zookeeper.RecoverableZooKeeper(93): Process identifier=master:45905 connecting to ZooKeeper ensemble=127.0.0.1:63943 2023-02-13 05:00:33,907 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:459050x0, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 05:00:33,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:45905-0x1004c0d4b820000 connected 2023-02-13 05:00:33,933 DEBUG [Listener at localhost/42165] zookeeper.ZKUtil(164): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 05:00:33,934 DEBUG [Listener at localhost/42165] zookeeper.ZKUtil(164): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 05:00:33,937 DEBUG [Listener at localhost/42165] zookeeper.ZKUtil(164): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 05:00:33,944 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45905 2023-02-13 05:00:33,945 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45905 2023-02-13 05:00:33,945 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45905 2023-02-13 05:00:33,945 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45905 2023-02-13 05:00:33,945 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45905 2023-02-13 05:00:33,951 INFO [Listener at localhost/42165] master.HMaster(439): hbase.rootdir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a, hbase.cluster.distributed=false 2023-02-13 05:00:34,016 INFO [Listener at localhost/42165] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-13 05:00:34,017 INFO [Listener at localhost/42165] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 05:00:34,017 INFO [Listener at localhost/42165] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 05:00:34,017 INFO [Listener at localhost/42165] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 05:00:34,017 INFO [Listener at localhost/42165] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 05:00:34,017 INFO [Listener at localhost/42165] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 05:00:34,022 INFO [Listener at localhost/42165] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 05:00:34,025 INFO [Listener at localhost/42165] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42589 2023-02-13 05:00:34,027 INFO [Listener at localhost/42165] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 05:00:34,032 INFO [Listener at localhost/42165] mob.MobFileCache(124): MobFileCache disabled 2023-02-13 05:00:34,034 INFO [Listener at localhost/42165] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 05:00:34,035 INFO [Listener at localhost/42165] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 05:00:34,037 INFO [Listener at localhost/42165] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42589 connecting to ZooKeeper ensemble=127.0.0.1:63943 2023-02-13 05:00:34,042 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:425890x0, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 05:00:34,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:42589-0x1004c0d4b820001 connected 2023-02-13 05:00:34,043 DEBUG [Listener at localhost/42165] zookeeper.ZKUtil(164): regionserver:42589-0x1004c0d4b820001, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 05:00:34,044 DEBUG [Listener at localhost/42165] zookeeper.ZKUtil(164): regionserver:42589-0x1004c0d4b820001, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 05:00:34,045 DEBUG [Listener at localhost/42165] zookeeper.ZKUtil(164): regionserver:42589-0x1004c0d4b820001, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 05:00:34,046 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42589 2023-02-13 05:00:34,046 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42589 2023-02-13 05:00:34,046 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42589 2023-02-13 05:00:34,047 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42589 2023-02-13 05:00:34,047 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42589 2023-02-13 05:00:34,059 INFO [Listener at localhost/42165] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-13 05:00:34,059 INFO [Listener at localhost/42165] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 05:00:34,060 INFO [Listener at localhost/42165] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 05:00:34,060 INFO [Listener at localhost/42165] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 05:00:34,060 INFO [Listener at localhost/42165] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 05:00:34,060 INFO [Listener at localhost/42165] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 05:00:34,061 INFO [Listener at localhost/42165] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 05:00:34,063 INFO [Listener at localhost/42165] ipc.NettyRpcServer(120): Bind to /172.31.14.131:39885 2023-02-13 05:00:34,063 INFO [Listener at localhost/42165] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 05:00:34,064 INFO [Listener at localhost/42165] mob.MobFileCache(124): MobFileCache disabled 2023-02-13 05:00:34,065 INFO [Listener at localhost/42165] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 05:00:34,066 INFO [Listener at localhost/42165] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 05:00:34,068 INFO [Listener at localhost/42165] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:39885 connecting to ZooKeeper ensemble=127.0.0.1:63943 2023-02-13 05:00:34,072 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:398850x0, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 05:00:34,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:39885-0x1004c0d4b820002 connected 2023-02-13 05:00:34,073 DEBUG [Listener at localhost/42165] zookeeper.ZKUtil(164): regionserver:39885-0x1004c0d4b820002, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 05:00:34,073 DEBUG [Listener at localhost/42165] zookeeper.ZKUtil(164): regionserver:39885-0x1004c0d4b820002, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 05:00:34,074 DEBUG [Listener at localhost/42165] zookeeper.ZKUtil(164): regionserver:39885-0x1004c0d4b820002, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 05:00:34,076 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=39885 2023-02-13 05:00:34,076 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=39885 2023-02-13 05:00:34,077 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=39885 2023-02-13 05:00:34,077 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=39885 2023-02-13 05:00:34,077 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=39885 2023-02-13 05:00:34,089 INFO [Listener at localhost/42165] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=18 2023-02-13 05:00:34,089 INFO [Listener at localhost/42165] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 05:00:34,090 INFO [Listener at localhost/42165] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-02-13 05:00:34,090 INFO [Listener at localhost/42165] ipc.RWQueueRpcExecutor(105): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-02-13 05:00:34,090 INFO [Listener at localhost/42165] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-02-13 05:00:34,090 INFO [Listener at localhost/42165] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-02-13 05:00:34,090 INFO [Listener at localhost/42165] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-02-13 05:00:34,091 INFO [Listener at localhost/42165] ipc.NettyRpcServer(120): Bind to /172.31.14.131:34363 2023-02-13 05:00:34,092 INFO [Listener at localhost/42165] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-02-13 05:00:34,093 INFO [Listener at localhost/42165] mob.MobFileCache(124): MobFileCache disabled 2023-02-13 05:00:34,093 INFO [Listener at localhost/42165] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 05:00:34,095 INFO [Listener at localhost/42165] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 05:00:34,097 INFO [Listener at localhost/42165] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:34363 connecting to ZooKeeper ensemble=127.0.0.1:63943 2023-02-13 05:00:34,101 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:343630x0, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-02-13 05:00:34,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:34363-0x1004c0d4b820003 connected 2023-02-13 05:00:34,102 DEBUG [Listener at localhost/42165] zookeeper.ZKUtil(164): regionserver:34363-0x1004c0d4b820003, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 05:00:34,103 DEBUG [Listener at localhost/42165] zookeeper.ZKUtil(164): regionserver:34363-0x1004c0d4b820003, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 05:00:34,103 DEBUG [Listener at localhost/42165] zookeeper.ZKUtil(164): regionserver:34363-0x1004c0d4b820003, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-02-13 05:00:34,104 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=34363 2023-02-13 05:00:34,104 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=34363 2023-02-13 05:00:34,106 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=34363 2023-02-13 05:00:34,106 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=34363 2023-02-13 05:00:34,106 DEBUG [Listener at localhost/42165] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=34363 2023-02-13 05:00:34,108 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2158): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,45905,1676264432900 2023-02-13 05:00:34,117 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 05:00:34,119 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,45905,1676264432900 2023-02-13 05:00:34,136 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 05:00:34,136 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:42589-0x1004c0d4b820001, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 05:00:34,136 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:39885-0x1004c0d4b820002, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 05:00:34,136 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:34363-0x1004c0d4b820003, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-02-13 05:00:34,136 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 05:00:34,138 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 05:00:34,139 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(224): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,45905,1676264432900 from backup master directory 2023-02-13 05:00:34,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-02-13 05:00:34,143 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,45905,1676264432900 2023-02-13 05:00:34,143 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-02-13 05:00:34,144 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 05:00:34,144 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(234): Registered as active master=jenkins-hbase4.apache.org,45905,1676264432900 2023-02-13 05:00:34,146 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-02-13 05:00:34,148 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-02-13 05:00:34,238 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(628): Created cluster ID file at hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/hbase.id with ID: 2b9ac89e-227f-4f1c-9dc1-aeaf3329cac8 2023-02-13 05:00:34,280 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-02-13 05:00:34,294 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 05:00:34,343 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x2d4a255d to 127.0.0.1:63943 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 05:00:34,377 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40f6743c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 05:00:34,401 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-02-13 05:00:34,403 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-02-13 05:00:34,421 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-02-13 05:00:34,421 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-02-13 05:00:34,423 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-13 05:00:34,428 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:849) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2178) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:523) at java.lang.Thread.run(Thread.java:750) 2023-02-13 05:00:34,429 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 05:00:34,461 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7689): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/MasterData/data/master/store-tmp 2023-02-13 05:00:34,524 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:34,525 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 05:00:34,525 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 05:00:34,525 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 05:00:34,525 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 05:00:34,525 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 05:00:34,525 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 05:00:34,525 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676264434524Disabling compacts and flushes for region at 1676264434525 (+1 ms)Disabling writes for close at 1676264434525Writing region close event to WAL at 1676264434525Closed at 1676264434525 2023-02-13 05:00:34,527 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/MasterData/WALs/jenkins-hbase4.apache.org,45905,1676264432900 2023-02-13 05:00:34,546 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45905%2C1676264432900, suffix=, logDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/MasterData/WALs/jenkins-hbase4.apache.org,45905,1676264432900, archiveDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/MasterData/oldWALs, maxLogs=10 2023-02-13 05:00:34,601 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35559,DS-d680ab8e-5805-4c53-a666-ef4d67b97527,DISK] 2023-02-13 05:00:34,601 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38213,DS-efdf400e-6075-477f-9e34-d836ad5a615a,DISK] 2023-02-13 05:00:34,601 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42015,DS-bb34b1c4-c7c8-49fb-bf39-0ff0ba335269,DISK] 2023-02-13 05:00:34,609 DEBUG [RS-EventLoopGroup-5-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-02-13 05:00:34,677 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/MasterData/WALs/jenkins-hbase4.apache.org,45905,1676264432900/jenkins-hbase4.apache.org%2C45905%2C1676264432900.1676264434556 2023-02-13 05:00:34,678 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:35559,DS-d680ab8e-5805-4c53-a666-ef4d67b97527,DISK], DatanodeInfoWithStorage[127.0.0.1:38213,DS-efdf400e-6075-477f-9e34-d836ad5a615a,DISK], DatanodeInfoWithStorage[127.0.0.1:42015,DS-bb34b1c4-c7c8-49fb-bf39-0ff0ba335269,DISK]] 2023-02-13 05:00:34,678 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7850): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-02-13 05:00:34,679 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(865): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:34,682 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7890): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 05:00:34,683 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7893): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 05:00:34,739 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-02-13 05:00:34,745 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-02-13 05:00:34,768 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-02-13 05:00:34,780 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:34,785 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 05:00:34,787 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-02-13 05:00:34,800 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1054): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-02-13 05:00:34,803 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 05:00:34,804 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1071): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70727564, jitterRate=0.053922832012176514}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:34,805 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(964): Region open journal for 1595e783b53d99cd5eef43b6debb2682: Writing region info on filesystem at 1676264434708Initializing all the Stores at 1676264434710 (+2 ms)Instantiating store for column family {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264434711 (+1 ms)Cleaning up temporary data from old regions at 1676264434792 (+81 ms)Cleaning up detritus from prior splits at 1676264434793 (+1 ms)Region opened successfully at 1676264434805 (+12 ms) 2023-02-13 05:00:34,806 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-02-13 05:00:34,824 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-02-13 05:00:34,825 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-02-13 05:00:34,827 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-02-13 05:00:34,828 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-02-13 05:00:34,860 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-02-13 05:00:34,860 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-02-13 05:00:34,884 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-02-13 05:00:34,890 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-02-13 05:00:34,914 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-02-13 05:00:34,917 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-02-13 05:00:34,918 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-02-13 05:00:34,924 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-02-13 05:00:34,928 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-02-13 05:00:34,930 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 05:00:34,932 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-02-13 05:00:34,933 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-02-13 05:00:34,946 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-02-13 05:00:34,951 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:39885-0x1004c0d4b820002, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 05:00:34,951 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:34363-0x1004c0d4b820003, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 05:00:34,951 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 05:00:34,951 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:42589-0x1004c0d4b820001, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-02-13 05:00:34,952 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 05:00:34,952 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(739): Active/primary master=jenkins-hbase4.apache.org,45905,1676264432900, sessionid=0x1004c0d4b820000, setting cluster-up flag (Was=false) 2023-02-13 05:00:34,967 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 05:00:34,974 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-02-13 05:00:34,976 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,45905,1676264432900 2023-02-13 05:00:34,980 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 05:00:34,991 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-02-13 05:00:34,993 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,45905,1676264432900 2023-02-13 05:00:34,995 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp 2023-02-13 05:00:35,010 INFO [RS:2;jenkins-hbase4:34363] regionserver.HRegionServer(952): ClusterId : 2b9ac89e-227f-4f1c-9dc1-aeaf3329cac8 2023-02-13 05:00:35,010 INFO [RS:0;jenkins-hbase4:42589] regionserver.HRegionServer(952): ClusterId : 2b9ac89e-227f-4f1c-9dc1-aeaf3329cac8 2023-02-13 05:00:35,010 INFO [RS:1;jenkins-hbase4:39885] regionserver.HRegionServer(952): ClusterId : 2b9ac89e-227f-4f1c-9dc1-aeaf3329cac8 2023-02-13 05:00:35,015 DEBUG [RS:1;jenkins-hbase4:39885] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 05:00:35,015 DEBUG [RS:0;jenkins-hbase4:42589] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 05:00:35,015 DEBUG [RS:2;jenkins-hbase4:34363] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-02-13 05:00:35,023 DEBUG [RS:0;jenkins-hbase4:42589] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 05:00:35,023 DEBUG [RS:1;jenkins-hbase4:39885] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 05:00:35,023 DEBUG [RS:2;jenkins-hbase4:34363] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-02-13 05:00:35,024 DEBUG [RS:1;jenkins-hbase4:39885] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 05:00:35,023 DEBUG [RS:0;jenkins-hbase4:42589] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 05:00:35,024 DEBUG [RS:2;jenkins-hbase4:34363] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-02-13 05:00:35,028 DEBUG [RS:1;jenkins-hbase4:39885] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 05:00:35,029 DEBUG [RS:2;jenkins-hbase4:34363] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 05:00:35,029 DEBUG [RS:1;jenkins-hbase4:39885] zookeeper.ReadOnlyZKClient(139): Connect 0x0e4c05a5 to 127.0.0.1:63943 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 05:00:35,029 DEBUG [RS:0;jenkins-hbase4:42589] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-02-13 05:00:35,031 DEBUG [RS:2;jenkins-hbase4:34363] zookeeper.ReadOnlyZKClient(139): Connect 0x5411a7c8 to 127.0.0.1:63943 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 05:00:35,031 DEBUG [RS:0;jenkins-hbase4:42589] zookeeper.ReadOnlyZKClient(139): Connect 0x1e714ffa to 127.0.0.1:63943 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 05:00:35,042 DEBUG [RS:1;jenkins-hbase4:39885] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8bfa1c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 05:00:35,042 DEBUG [RS:1;jenkins-hbase4:39885] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b46ed5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 05:00:35,045 DEBUG [RS:2;jenkins-hbase4:34363] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ad6533a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 05:00:35,046 DEBUG [RS:2;jenkins-hbase4:34363] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ee9ccb8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 05:00:35,046 DEBUG [RS:0;jenkins-hbase4:42589] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10b6a244, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 05:00:35,047 DEBUG [RS:0;jenkins-hbase4:42589] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1489248a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 05:00:35,067 DEBUG [RS:1;jenkins-hbase4:39885] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:1;jenkins-hbase4:39885 2023-02-13 05:00:35,067 DEBUG [RS:2;jenkins-hbase4:34363] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:2;jenkins-hbase4:34363 2023-02-13 05:00:35,071 DEBUG [RS:0;jenkins-hbase4:42589] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:42589 2023-02-13 05:00:35,072 INFO [RS:2;jenkins-hbase4:34363] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 05:00:35,072 INFO [RS:1;jenkins-hbase4:39885] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 05:00:35,072 INFO [RS:1;jenkins-hbase4:39885] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 05:00:35,072 INFO [RS:0;jenkins-hbase4:42589] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-02-13 05:00:35,072 DEBUG [RS:1;jenkins-hbase4:39885] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 05:00:35,072 INFO [RS:2;jenkins-hbase4:34363] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 05:00:35,072 INFO [RS:0;jenkins-hbase4:42589] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-02-13 05:00:35,073 DEBUG [RS:2;jenkins-hbase4:34363] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 05:00:35,073 DEBUG [RS:0;jenkins-hbase4:42589] regionserver.HRegionServer(1023): About to register with Master. 2023-02-13 05:00:35,076 INFO [RS:2;jenkins-hbase4:34363] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,45905,1676264432900 with isa=jenkins-hbase4.apache.org/172.31.14.131:34363, startcode=1676264434089 2023-02-13 05:00:35,076 INFO [RS:1;jenkins-hbase4:39885] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,45905,1676264432900 with isa=jenkins-hbase4.apache.org/172.31.14.131:39885, startcode=1676264434059 2023-02-13 05:00:35,076 INFO [RS:0;jenkins-hbase4:42589] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,45905,1676264432900 with isa=jenkins-hbase4.apache.org/172.31.14.131:42589, startcode=1676264434015 2023-02-13 05:00:35,094 DEBUG [RS:1;jenkins-hbase4:39885] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 05:00:35,094 DEBUG [RS:2;jenkins-hbase4:34363] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 05:00:35,094 DEBUG [RS:0;jenkins-hbase4:42589] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-02-13 05:00:35,139 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-02-13 05:00:35,149 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 05:00:35,149 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 05:00:35,149 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 05:00:35,150 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-02-13 05:00:35,150 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-02-13 05:00:35,150 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,150 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 05:00:35,150 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,151 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33727, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.1 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 05:00:35,151 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47699, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.2 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 05:00:35,151 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46011, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-02-13 05:00:35,152 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1676264465152 2023-02-13 05:00:35,155 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-02-13 05:00:35,159 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 05:00:35,159 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-02-13 05:00:35,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45905] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:35,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45905] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:35,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2815) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:577) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:15952) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:35,166 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 05:00:35,169 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-02-13 05:00:35,177 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-02-13 05:00:35,178 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-02-13 05:00:35,180 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-02-13 05:00:35,180 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-02-13 05:00:35,184 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,187 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-02-13 05:00:35,189 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-02-13 05:00:35,189 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-02-13 05:00:35,194 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-02-13 05:00:35,195 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-02-13 05:00:35,199 DEBUG [RS:1;jenkins-hbase4:39885] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 05:00:35,200 DEBUG [RS:2;jenkins-hbase4:34363] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 05:00:35,200 WARN [RS:1;jenkins-hbase4:39885] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-13 05:00:35,200 DEBUG [RS:0;jenkins-hbase4:42589] regionserver.HRegionServer(2829): Master is not running yet 2023-02-13 05:00:35,199 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676264435198,5,FailOnTimeoutGroup] 2023-02-13 05:00:35,201 WARN [RS:0;jenkins-hbase4:42589] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-13 05:00:35,201 WARN [RS:2;jenkins-hbase4:34363] regionserver.HRegionServer(1031): reportForDuty failed; sleeping 100 ms and then retrying. 2023-02-13 05:00:35,203 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676264435203,5,FailOnTimeoutGroup] 2023-02-13 05:00:35,203 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,204 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1451): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-02-13 05:00:35,207 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,207 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,213 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 05:00:35,214 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-02-13 05:00:35,215 INFO [PEWorker-1] regionserver.HRegion(7671): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a 2023-02-13 05:00:35,235 DEBUG [PEWorker-1] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:35,237 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 05:00:35,239 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/info 2023-02-13 05:00:35,240 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 05:00:35,241 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:35,241 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 05:00:35,243 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/rep_barrier 2023-02-13 05:00:35,244 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 05:00:35,245 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:35,245 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 05:00:35,247 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/table 2023-02-13 05:00:35,247 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 05:00:35,248 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:35,249 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740 2023-02-13 05:00:35,250 DEBUG [PEWorker-1] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740 2023-02-13 05:00:35,253 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 05:00:35,255 DEBUG [PEWorker-1] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 05:00:35,259 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 05:00:35,259 INFO [PEWorker-1] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75123668, jitterRate=0.11942988634109497}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 05:00:35,259 DEBUG [PEWorker-1] regionserver.HRegion(964): Region open journal for 1588230740: Writing region info on filesystem at 1676264435235Initializing all the Stores at 1676264435236 (+1 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676264435236Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264435237 (+1 ms)Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676264435237Cleaning up temporary data from old regions at 1676264435252 (+15 ms)Cleaning up detritus from prior splits at 1676264435252Region opened successfully at 1676264435259 (+7 ms) 2023-02-13 05:00:35,260 DEBUG [PEWorker-1] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 05:00:35,260 INFO [PEWorker-1] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 05:00:35,260 DEBUG [PEWorker-1] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 05:00:35,260 DEBUG [PEWorker-1] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 05:00:35,260 DEBUG [PEWorker-1] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 05:00:35,261 INFO [PEWorker-1] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 05:00:35,261 DEBUG [PEWorker-1] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676264435260Disabling compacts and flushes for region at 1676264435260Disabling writes for close at 1676264435260Writing region close event to WAL at 1676264435261 (+1 ms)Closed at 1676264435261 2023-02-13 05:00:35,266 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-02-13 05:00:35,266 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-02-13 05:00:35,273 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-02-13 05:00:35,285 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-02-13 05:00:35,288 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-02-13 05:00:35,302 INFO [RS:1;jenkins-hbase4:39885] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,45905,1676264432900 with isa=jenkins-hbase4.apache.org/172.31.14.131:39885, startcode=1676264434059 2023-02-13 05:00:35,304 INFO [RS:2;jenkins-hbase4:34363] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,45905,1676264432900 with isa=jenkins-hbase4.apache.org/172.31.14.131:34363, startcode=1676264434089 2023-02-13 05:00:35,304 INFO [RS:0;jenkins-hbase4:42589] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,45905,1676264432900 with isa=jenkins-hbase4.apache.org/172.31.14.131:42589, startcode=1676264434015 2023-02-13 05:00:35,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45905] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:35,308 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45905] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:35,309 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:35,315 DEBUG [RS:0;jenkins-hbase4:42589] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a 2023-02-13 05:00:35,315 DEBUG [RS:2;jenkins-hbase4:34363] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a 2023-02-13 05:00:35,315 DEBUG [RS:0;jenkins-hbase4:42589] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:36489 2023-02-13 05:00:35,316 DEBUG [RS:2;jenkins-hbase4:34363] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:36489 2023-02-13 05:00:35,316 DEBUG [RS:0;jenkins-hbase4:42589] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 05:00:35,316 DEBUG [RS:2;jenkins-hbase4:34363] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 05:00:35,316 DEBUG [RS:1;jenkins-hbase4:39885] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a 2023-02-13 05:00:35,316 DEBUG [RS:1;jenkins-hbase4:39885] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:36489 2023-02-13 05:00:35,316 DEBUG [RS:1;jenkins-hbase4:39885] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-02-13 05:00:35,325 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 05:00:35,325 DEBUG [RS:1;jenkins-hbase4:39885] zookeeper.ZKUtil(162): regionserver:39885-0x1004c0d4b820002, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:35,325 DEBUG [RS:0;jenkins-hbase4:42589] zookeeper.ZKUtil(162): regionserver:42589-0x1004c0d4b820001, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:35,325 WARN [RS:1;jenkins-hbase4:39885] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 05:00:35,325 DEBUG [RS:2;jenkins-hbase4:34363] zookeeper.ZKUtil(162): regionserver:34363-0x1004c0d4b820003, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:35,326 INFO [RS:1;jenkins-hbase4:39885] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 05:00:35,326 WARN [RS:2;jenkins-hbase4:34363] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 05:00:35,325 WARN [RS:0;jenkins-hbase4:42589] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-02-13 05:00:35,326 INFO [RS:2;jenkins-hbase4:34363] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 05:00:35,326 DEBUG [RS:1;jenkins-hbase4:39885] regionserver.HRegionServer(1947): logDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/WALs/jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:35,327 DEBUG [RS:2;jenkins-hbase4:34363] regionserver.HRegionServer(1947): logDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/WALs/jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:35,327 INFO [RS:0;jenkins-hbase4:42589] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 05:00:35,327 DEBUG [RS:0;jenkins-hbase4:42589] regionserver.HRegionServer(1947): logDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/WALs/jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:35,328 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,42589,1676264434015] 2023-02-13 05:00:35,328 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,34363,1676264434089] 2023-02-13 05:00:35,328 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,39885,1676264434059] 2023-02-13 05:00:35,338 DEBUG [RS:2;jenkins-hbase4:34363] zookeeper.ZKUtil(162): regionserver:34363-0x1004c0d4b820003, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:35,338 DEBUG [RS:0;jenkins-hbase4:42589] zookeeper.ZKUtil(162): regionserver:42589-0x1004c0d4b820001, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:35,338 DEBUG [RS:1;jenkins-hbase4:39885] zookeeper.ZKUtil(162): regionserver:39885-0x1004c0d4b820002, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:35,339 DEBUG [RS:2;jenkins-hbase4:34363] zookeeper.ZKUtil(162): regionserver:34363-0x1004c0d4b820003, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:35,339 DEBUG [RS:0;jenkins-hbase4:42589] zookeeper.ZKUtil(162): regionserver:42589-0x1004c0d4b820001, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:35,339 DEBUG [RS:1;jenkins-hbase4:39885] zookeeper.ZKUtil(162): regionserver:39885-0x1004c0d4b820002, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:35,339 DEBUG [RS:2;jenkins-hbase4:34363] zookeeper.ZKUtil(162): regionserver:34363-0x1004c0d4b820003, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:35,339 DEBUG [RS:0;jenkins-hbase4:42589] zookeeper.ZKUtil(162): regionserver:42589-0x1004c0d4b820001, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:35,339 DEBUG [RS:1;jenkins-hbase4:39885] zookeeper.ZKUtil(162): regionserver:39885-0x1004c0d4b820002, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:35,349 DEBUG [RS:1;jenkins-hbase4:39885] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 05:00:35,349 DEBUG [RS:2;jenkins-hbase4:34363] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 05:00:35,349 DEBUG [RS:0;jenkins-hbase4:42589] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-02-13 05:00:35,358 INFO [RS:2;jenkins-hbase4:34363] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 05:00:35,358 INFO [RS:1;jenkins-hbase4:39885] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 05:00:35,358 INFO [RS:0;jenkins-hbase4:42589] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-02-13 05:00:35,378 INFO [RS:0;jenkins-hbase4:42589] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 05:00:35,378 INFO [RS:2;jenkins-hbase4:34363] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 05:00:35,378 INFO [RS:1;jenkins-hbase4:39885] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-02-13 05:00:35,382 INFO [RS:1;jenkins-hbase4:39885] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 05:00:35,382 INFO [RS:2;jenkins-hbase4:34363] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 05:00:35,382 INFO [RS:1;jenkins-hbase4:39885] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,382 INFO [RS:0;jenkins-hbase4:42589] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-02-13 05:00:35,382 INFO [RS:2;jenkins-hbase4:34363] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,383 INFO [RS:0;jenkins-hbase4:42589] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,383 INFO [RS:1;jenkins-hbase4:39885] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 05:00:35,383 INFO [RS:2;jenkins-hbase4:34363] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 05:00:35,383 INFO [RS:0;jenkins-hbase4:42589] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-02-13 05:00:35,391 INFO [RS:0;jenkins-hbase4:42589] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,391 INFO [RS:1;jenkins-hbase4:39885] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,391 INFO [RS:2;jenkins-hbase4:34363] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,391 DEBUG [RS:0;jenkins-hbase4:42589] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,391 DEBUG [RS:1;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,391 DEBUG [RS:0;jenkins-hbase4:42589] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,392 DEBUG [RS:1;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,392 DEBUG [RS:0;jenkins-hbase4:42589] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,392 DEBUG [RS:1;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,392 DEBUG [RS:0;jenkins-hbase4:42589] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,392 DEBUG [RS:1;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,392 DEBUG [RS:0;jenkins-hbase4:42589] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,392 DEBUG [RS:1;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,391 DEBUG [RS:2;jenkins-hbase4:34363] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,392 DEBUG [RS:1;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 05:00:35,392 DEBUG [RS:2;jenkins-hbase4:34363] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,392 DEBUG [RS:1;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,392 DEBUG [RS:2;jenkins-hbase4:34363] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,392 DEBUG [RS:1;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,392 DEBUG [RS:2;jenkins-hbase4:34363] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,392 DEBUG [RS:1;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,392 DEBUG [RS:0;jenkins-hbase4:42589] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 05:00:35,392 DEBUG [RS:1;jenkins-hbase4:39885] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,392 DEBUG [RS:2;jenkins-hbase4:34363] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,393 DEBUG [RS:0;jenkins-hbase4:42589] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,393 DEBUG [RS:2;jenkins-hbase4:34363] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-02-13 05:00:35,393 DEBUG [RS:0;jenkins-hbase4:42589] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,393 DEBUG [RS:2;jenkins-hbase4:34363] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,393 DEBUG [RS:0;jenkins-hbase4:42589] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,393 INFO [RS:1;jenkins-hbase4:39885] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,394 DEBUG [RS:0;jenkins-hbase4:42589] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,393 DEBUG [RS:2;jenkins-hbase4:34363] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,394 INFO [RS:1;jenkins-hbase4:39885] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,394 DEBUG [RS:2;jenkins-hbase4:34363] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,394 INFO [RS:1;jenkins-hbase4:39885] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,394 DEBUG [RS:2;jenkins-hbase4:34363] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-02-13 05:00:35,395 INFO [RS:0;jenkins-hbase4:42589] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,395 INFO [RS:0;jenkins-hbase4:42589] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,395 INFO [RS:2;jenkins-hbase4:34363] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,395 INFO [RS:0;jenkins-hbase4:42589] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,395 INFO [RS:2;jenkins-hbase4:34363] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,395 INFO [RS:2;jenkins-hbase4:34363] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,410 INFO [RS:2;jenkins-hbase4:34363] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 05:00:35,411 INFO [RS:1;jenkins-hbase4:39885] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 05:00:35,411 INFO [RS:0;jenkins-hbase4:42589] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-02-13 05:00:35,413 INFO [RS:2;jenkins-hbase4:34363] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,34363,1676264434089-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,413 INFO [RS:0;jenkins-hbase4:42589] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42589,1676264434015-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,413 INFO [RS:1;jenkins-hbase4:39885] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,39885,1676264434059-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,431 INFO [RS:2;jenkins-hbase4:34363] regionserver.Replication(203): jenkins-hbase4.apache.org,34363,1676264434089 started 2023-02-13 05:00:35,431 INFO [RS:1;jenkins-hbase4:39885] regionserver.Replication(203): jenkins-hbase4.apache.org,39885,1676264434059 started 2023-02-13 05:00:35,431 INFO [RS:2;jenkins-hbase4:34363] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,34363,1676264434089, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:34363, sessionid=0x1004c0d4b820003 2023-02-13 05:00:35,431 INFO [RS:1;jenkins-hbase4:39885] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,39885,1676264434059, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:39885, sessionid=0x1004c0d4b820002 2023-02-13 05:00:35,431 DEBUG [RS:1;jenkins-hbase4:39885] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 05:00:35,431 DEBUG [RS:2;jenkins-hbase4:34363] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 05:00:35,432 DEBUG [RS:1;jenkins-hbase4:39885] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:35,432 DEBUG [RS:2;jenkins-hbase4:34363] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:35,432 DEBUG [RS:1;jenkins-hbase4:39885] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39885,1676264434059' 2023-02-13 05:00:35,432 DEBUG [RS:2;jenkins-hbase4:34363] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34363,1676264434089' 2023-02-13 05:00:35,433 DEBUG [RS:2;jenkins-hbase4:34363] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 05:00:35,433 DEBUG [RS:1;jenkins-hbase4:39885] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 05:00:35,434 DEBUG [RS:1;jenkins-hbase4:39885] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 05:00:35,434 DEBUG [RS:2;jenkins-hbase4:34363] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 05:00:35,434 INFO [RS:0;jenkins-hbase4:42589] regionserver.Replication(203): jenkins-hbase4.apache.org,42589,1676264434015 started 2023-02-13 05:00:35,434 INFO [RS:0;jenkins-hbase4:42589] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,42589,1676264434015, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:42589, sessionid=0x1004c0d4b820001 2023-02-13 05:00:35,434 DEBUG [RS:0;jenkins-hbase4:42589] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-02-13 05:00:35,434 DEBUG [RS:2;jenkins-hbase4:34363] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 05:00:35,434 DEBUG [RS:1;jenkins-hbase4:39885] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 05:00:35,434 DEBUG [RS:2;jenkins-hbase4:34363] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 05:00:35,434 DEBUG [RS:0;jenkins-hbase4:42589] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:35,434 DEBUG [RS:2;jenkins-hbase4:34363] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:35,434 DEBUG [RS:1;jenkins-hbase4:39885] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 05:00:35,435 DEBUG [RS:2;jenkins-hbase4:34363] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,34363,1676264434089' 2023-02-13 05:00:35,435 DEBUG [RS:2;jenkins-hbase4:34363] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 05:00:35,434 DEBUG [RS:0;jenkins-hbase4:42589] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42589,1676264434015' 2023-02-13 05:00:35,435 DEBUG [RS:1;jenkins-hbase4:39885] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:35,435 DEBUG [RS:0;jenkins-hbase4:42589] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-02-13 05:00:35,435 DEBUG [RS:1;jenkins-hbase4:39885] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,39885,1676264434059' 2023-02-13 05:00:35,436 DEBUG [RS:2;jenkins-hbase4:34363] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 05:00:35,436 DEBUG [RS:1;jenkins-hbase4:39885] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 05:00:35,436 DEBUG [RS:0;jenkins-hbase4:42589] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-02-13 05:00:35,436 DEBUG [RS:2;jenkins-hbase4:34363] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 05:00:35,436 INFO [RS:2;jenkins-hbase4:34363] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 05:00:35,436 DEBUG [RS:1;jenkins-hbase4:39885] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 05:00:35,436 INFO [RS:2;jenkins-hbase4:34363] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 05:00:35,436 DEBUG [RS:0;jenkins-hbase4:42589] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-02-13 05:00:35,437 DEBUG [RS:0;jenkins-hbase4:42589] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-02-13 05:00:35,437 DEBUG [RS:0;jenkins-hbase4:42589] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:35,437 DEBUG [RS:0;jenkins-hbase4:42589] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42589,1676264434015' 2023-02-13 05:00:35,437 DEBUG [RS:0;jenkins-hbase4:42589] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-02-13 05:00:35,437 DEBUG [RS:1;jenkins-hbase4:39885] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 05:00:35,437 INFO [RS:1;jenkins-hbase4:39885] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 05:00:35,438 DEBUG [RS:0;jenkins-hbase4:42589] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-02-13 05:00:35,438 INFO [RS:1;jenkins-hbase4:39885] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 05:00:35,440 DEBUG [RS:0;jenkins-hbase4:42589] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-02-13 05:00:35,440 INFO [RS:0;jenkins-hbase4:42589] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-02-13 05:00:35,440 INFO [RS:0;jenkins-hbase4:42589] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-02-13 05:00:35,441 DEBUG [jenkins-hbase4:45905] assignment.AssignmentManager(2178): Processing assignQueue; systemServersCount=3, allServersCount=3 2023-02-13 05:00:35,444 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 05:00:35,449 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 05:00:35,449 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 05:00:35,449 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 05:00:35,449 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 05:00:35,452 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39885,1676264434059, state=OPENING 2023-02-13 05:00:35,462 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-02-13 05:00:35,464 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 05:00:35,464 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 05:00:35,468 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:35,547 INFO [RS:2;jenkins-hbase4:34363] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C34363%2C1676264434089, suffix=, logDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/WALs/jenkins-hbase4.apache.org,34363,1676264434089, archiveDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/oldWALs, maxLogs=32 2023-02-13 05:00:35,547 INFO [RS:0;jenkins-hbase4:42589] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42589%2C1676264434015, suffix=, logDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/WALs/jenkins-hbase4.apache.org,42589,1676264434015, archiveDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/oldWALs, maxLogs=32 2023-02-13 05:00:35,547 INFO [RS:1;jenkins-hbase4:39885] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39885%2C1676264434059, suffix=, logDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/WALs/jenkins-hbase4.apache.org,39885,1676264434059, archiveDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/oldWALs, maxLogs=32 2023-02-13 05:00:35,576 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38213,DS-efdf400e-6075-477f-9e34-d836ad5a615a,DISK] 2023-02-13 05:00:35,576 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42015,DS-bb34b1c4-c7c8-49fb-bf39-0ff0ba335269,DISK] 2023-02-13 05:00:35,576 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38213,DS-efdf400e-6075-477f-9e34-d836ad5a615a,DISK] 2023-02-13 05:00:35,576 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35559,DS-d680ab8e-5805-4c53-a666-ef4d67b97527,DISK] 2023-02-13 05:00:35,579 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35559,DS-d680ab8e-5805-4c53-a666-ef4d67b97527,DISK] 2023-02-13 05:00:35,579 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42015,DS-bb34b1c4-c7c8-49fb-bf39-0ff0ba335269,DISK] 2023-02-13 05:00:35,593 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35559,DS-d680ab8e-5805-4c53-a666-ef4d67b97527,DISK] 2023-02-13 05:00:35,593 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38213,DS-efdf400e-6075-477f-9e34-d836ad5a615a,DISK] 2023-02-13 05:00:35,593 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42015,DS-bb34b1c4-c7c8-49fb-bf39-0ff0ba335269,DISK] 2023-02-13 05:00:35,594 INFO [RS:1;jenkins-hbase4:39885] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/WALs/jenkins-hbase4.apache.org,39885,1676264434059/jenkins-hbase4.apache.org%2C39885%2C1676264434059.1676264435552 2023-02-13 05:00:35,609 DEBUG [RS:1;jenkins-hbase4:39885] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42015,DS-bb34b1c4-c7c8-49fb-bf39-0ff0ba335269,DISK], DatanodeInfoWithStorage[127.0.0.1:38213,DS-efdf400e-6075-477f-9e34-d836ad5a615a,DISK], DatanodeInfoWithStorage[127.0.0.1:35559,DS-d680ab8e-5805-4c53-a666-ef4d67b97527,DISK]] 2023-02-13 05:00:35,610 INFO [RS:2;jenkins-hbase4:34363] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/WALs/jenkins-hbase4.apache.org,34363,1676264434089/jenkins-hbase4.apache.org%2C34363%2C1676264434089.1676264435552 2023-02-13 05:00:35,610 DEBUG [RS:2;jenkins-hbase4:34363] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38213,DS-efdf400e-6075-477f-9e34-d836ad5a615a,DISK], DatanodeInfoWithStorage[127.0.0.1:35559,DS-d680ab8e-5805-4c53-a666-ef4d67b97527,DISK], DatanodeInfoWithStorage[127.0.0.1:42015,DS-bb34b1c4-c7c8-49fb-bf39-0ff0ba335269,DISK]] 2023-02-13 05:00:35,610 INFO [RS:0;jenkins-hbase4:42589] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/WALs/jenkins-hbase4.apache.org,42589,1676264434015/jenkins-hbase4.apache.org%2C42589%2C1676264434015.1676264435552 2023-02-13 05:00:35,611 DEBUG [RS:0;jenkins-hbase4:42589] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38213,DS-efdf400e-6075-477f-9e34-d836ad5a615a,DISK], DatanodeInfoWithStorage[127.0.0.1:35559,DS-d680ab8e-5805-4c53-a666-ef4d67b97527,DISK], DatanodeInfoWithStorage[127.0.0.1:42015,DS-bb34b1c4-c7c8-49fb-bf39-0ff0ba335269,DISK]] 2023-02-13 05:00:35,653 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:35,656 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 05:00:35,659 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43826, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 05:00:35,671 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-02-13 05:00:35,671 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-02-13 05:00:35,674 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(464): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C39885%2C1676264434059.meta, suffix=.meta, logDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/WALs/jenkins-hbase4.apache.org,39885,1676264434059, archiveDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/oldWALs, maxLogs=32 2023-02-13 05:00:35,696 DEBUG [RS-EventLoopGroup-5-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:38213,DS-efdf400e-6075-477f-9e34-d836ad5a615a,DISK] 2023-02-13 05:00:35,697 DEBUG [RS-EventLoopGroup-5-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42015,DS-bb34b1c4-c7c8-49fb-bf39-0ff0ba335269,DISK] 2023-02-13 05:00:35,697 DEBUG [RS-EventLoopGroup-5-3] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:35559,DS-d680ab8e-5805-4c53-a666-ef4d67b97527,DISK] 2023-02-13 05:00:35,703 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(758): New WAL /user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/WALs/jenkins-hbase4.apache.org,39885,1676264434059/jenkins-hbase4.apache.org%2C39885%2C1676264434059.meta.1676264435676.meta 2023-02-13 05:00:35,703 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(839): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:38213,DS-efdf400e-6075-477f-9e34-d836ad5a615a,DISK], DatanodeInfoWithStorage[127.0.0.1:42015,DS-bb34b1c4-c7c8-49fb-bf39-0ff0ba335269,DISK], DatanodeInfoWithStorage[127.0.0.1:35559,DS-d680ab8e-5805-4c53-a666-ef4d67b97527,DISK]] 2023-02-13 05:00:35,703 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-02-13 05:00:35,705 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-02-13 05:00:35,722 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8546): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-02-13 05:00:35,727 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-02-13 05:00:35,732 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-02-13 05:00:35,733 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:35,734 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 1588230740 2023-02-13 05:00:35,734 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 1588230740 2023-02-13 05:00:35,738 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-02-13 05:00:35,740 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/info 2023-02-13 05:00:35,740 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/info 2023-02-13 05:00:35,741 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-02-13 05:00:35,741 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:35,742 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-02-13 05:00:35,743 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/rep_barrier 2023-02-13 05:00:35,743 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/rep_barrier 2023-02-13 05:00:35,744 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-02-13 05:00:35,745 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:35,745 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-02-13 05:00:35,746 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/table 2023-02-13 05:00:35,746 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/table 2023-02-13 05:00:35,747 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-02-13 05:00:35,747 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:35,749 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740 2023-02-13 05:00:35,752 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740 2023-02-13 05:00:35,755 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (42.7 M)) instead. 2023-02-13 05:00:35,757 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 1588230740 2023-02-13 05:00:35,759 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 1588230740; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68755236, jitterRate=0.024532854557037354}}}, FlushLargeStoresPolicy{flushSizeLowerBound=44739242} 2023-02-13 05:00:35,759 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 1588230740: Running coprocessor pre-open hook at 1676264435734Writing region info on filesystem at 1676264435735 (+1 ms)Initializing all the Stores at 1676264435738 (+3 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676264435738Instantiating store for column family {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} at 1676264435738Instantiating store for column family {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676264435738Cleaning up temporary data from old regions at 1676264435753 (+15 ms)Cleaning up detritus from prior splits at 1676264435754 (+1 ms)Running coprocessor post-open hooks at 1676264435759 (+5 ms)Region opened successfully at 1676264435759 2023-02-13 05:00:35,769 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1676264435646 2023-02-13 05:00:35,786 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-02-13 05:00:35,787 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-02-13 05:00:35,788 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,39885,1676264434059, state=OPEN 2023-02-13 05:00:35,790 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-02-13 05:00:35,791 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-02-13 05:00:35,795 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-02-13 05:00:35,795 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,39885,1676264434059 in 322 msec 2023-02-13 05:00:35,799 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-02-13 05:00:35,800 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 522 msec 2023-02-13 05:00:35,805 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 746 msec 2023-02-13 05:00:35,805 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(946): Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1676264434132, completionTime=-1 2023-02-13 05:00:35,805 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=3; waited=0ms, expected min=3 server(s), max=3 server(s), master is running 2023-02-13 05:00:35,805 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1519): Joining cluster... 2023-02-13 05:00:35,871 DEBUG [hconnection-0x10a3a304-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:35,874 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43842, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:35,891 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1531): Number of RegionServers=3 2023-02-13 05:00:35,891 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1676264495891 2023-02-13 05:00:35,891 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1676264555891 2023-02-13 05:00:35,891 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1538): Joined the cluster in 85 msec 2023-02-13 05:00:35,914 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45905,1676264432900-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,914 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45905,1676264432900-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,915 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45905,1676264432900-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,916 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:45905, period=300000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,917 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-02-13 05:00:35,926 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-02-13 05:00:35,935 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-02-13 05:00:35,936 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2138): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-02-13 05:00:35,947 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-02-13 05:00:35,950 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 05:00:35,952 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 05:00:35,974 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/hbase/namespace/03df2a43656204957e15d49e0f693da9 2023-02-13 05:00:35,977 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/hbase/namespace/03df2a43656204957e15d49e0f693da9 empty. 2023-02-13 05:00:35,977 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/hbase/namespace/03df2a43656204957e15d49e0f693da9 2023-02-13 05:00:35,978 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-02-13 05:00:36,013 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-02-13 05:00:36,015 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7671): creating {ENCODED => 03df2a43656204957e15d49e0f693da9, NAME => 'hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp 2023-02-13 05:00:36,034 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:36,035 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1603): Closing 03df2a43656204957e15d49e0f693da9, disabling compactions & flushes 2023-02-13 05:00:36,035 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9. 2023-02-13 05:00:36,035 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9. 2023-02-13 05:00:36,035 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9. after waiting 0 ms 2023-02-13 05:00:36,035 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9. 2023-02-13 05:00:36,035 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9. 2023-02-13 05:00:36,035 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1557): Region close journal for 03df2a43656204957e15d49e0f693da9: Waiting for close lock at 1676264436035Disabling compacts and flushes for region at 1676264436035Disabling writes for close at 1676264436035Writing region close event to WAL at 1676264436035Closed at 1676264436035 2023-02-13 05:00:36,040 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 05:00:36,056 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676264436043"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264436043"}]},"ts":"1676264436043"} 2023-02-13 05:00:36,083 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-02-13 05:00:36,085 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 05:00:36,090 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264436085"}]},"ts":"1676264436085"} 2023-02-13 05:00:36,094 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-02-13 05:00:36,098 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 05:00:36,099 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 05:00:36,099 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 05:00:36,099 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 05:00:36,099 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 05:00:36,102 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=03df2a43656204957e15d49e0f693da9, ASSIGN}] 2023-02-13 05:00:36,105 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=03df2a43656204957e15d49e0f693da9, ASSIGN 2023-02-13 05:00:36,106 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=03df2a43656204957e15d49e0f693da9, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34363,1676264434089; forceNewPlan=false, retain=false 2023-02-13 05:00:36,258 INFO [jenkins-hbase4:45905] balancer.BaseLoadBalancer(1545): Reassigned 1 regions. 1 retained the pre-restart assignment. 2023-02-13 05:00:36,259 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=03df2a43656204957e15d49e0f693da9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:36,259 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676264436259"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264436259"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264436259"}]},"ts":"1676264436259"} 2023-02-13 05:00:36,263 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 03df2a43656204957e15d49e0f693da9, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:36,418 DEBUG [RSProcedureDispatcher-pool-1] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:36,418 DEBUG [RSProcedureDispatcher-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 05:00:36,422 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44898, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 05:00:36,428 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9. 2023-02-13 05:00:36,430 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 03df2a43656204957e15d49e0f693da9, NAME => 'hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9.', STARTKEY => '', ENDKEY => ''} 2023-02-13 05:00:36,431 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 03df2a43656204957e15d49e0f693da9 2023-02-13 05:00:36,431 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:36,431 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 03df2a43656204957e15d49e0f693da9 2023-02-13 05:00:36,431 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 03df2a43656204957e15d49e0f693da9 2023-02-13 05:00:36,433 INFO [StoreOpener-03df2a43656204957e15d49e0f693da9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 03df2a43656204957e15d49e0f693da9 2023-02-13 05:00:36,435 DEBUG [StoreOpener-03df2a43656204957e15d49e0f693da9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/namespace/03df2a43656204957e15d49e0f693da9/info 2023-02-13 05:00:36,435 DEBUG [StoreOpener-03df2a43656204957e15d49e0f693da9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/namespace/03df2a43656204957e15d49e0f693da9/info 2023-02-13 05:00:36,436 INFO [StoreOpener-03df2a43656204957e15d49e0f693da9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 03df2a43656204957e15d49e0f693da9 columnFamilyName info 2023-02-13 05:00:36,437 INFO [StoreOpener-03df2a43656204957e15d49e0f693da9-1] regionserver.HStore(310): Store=03df2a43656204957e15d49e0f693da9/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:36,439 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/namespace/03df2a43656204957e15d49e0f693da9 2023-02-13 05:00:36,440 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/namespace/03df2a43656204957e15d49e0f693da9 2023-02-13 05:00:36,444 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 03df2a43656204957e15d49e0f693da9 2023-02-13 05:00:36,448 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/namespace/03df2a43656204957e15d49e0f693da9/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 05:00:36,449 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 03df2a43656204957e15d49e0f693da9; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61597587, jitterRate=-0.08212442696094513}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:36,449 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 03df2a43656204957e15d49e0f693da9: Running coprocessor pre-open hook at 1676264436431Writing region info on filesystem at 1676264436431Initializing all the Stores at 1676264436433 (+2 ms)Instantiating store for column family {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} at 1676264436433Cleaning up temporary data from old regions at 1676264436440 (+7 ms)Cleaning up detritus from prior splits at 1676264436441 (+1 ms)Running coprocessor post-open hooks at 1676264436449 (+8 ms)Region opened successfully at 1676264436449 2023-02-13 05:00:36,452 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9., pid=6, masterSystemTime=1676264436418 2023-02-13 05:00:36,456 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9. 2023-02-13 05:00:36,456 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9. 2023-02-13 05:00:36,458 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=03df2a43656204957e15d49e0f693da9, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:36,459 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1676264436457"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264436457"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264436457"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264436457"}]},"ts":"1676264436457"} 2023-02-13 05:00:36,465 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-02-13 05:00:36,465 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 03df2a43656204957e15d49e0f693da9, server=jenkins-hbase4.apache.org,34363,1676264434089 in 198 msec 2023-02-13 05:00:36,469 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-02-13 05:00:36,469 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=03df2a43656204957e15d49e0f693da9, ASSIGN in 363 msec 2023-02-13 05:00:36,470 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 05:00:36,470 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264436470"}]},"ts":"1676264436470"} 2023-02-13 05:00:36,473 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-02-13 05:00:36,476 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 05:00:36,479 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 539 msec 2023-02-13 05:00:36,550 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-02-13 05:00:36,552 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-02-13 05:00:36,552 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 05:00:36,573 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:36,575 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44902, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:36,587 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-02-13 05:00:36,605 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 05:00:36,612 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 31 msec 2023-02-13 05:00:36,621 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-02-13 05:00:36,632 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-02-13 05:00:36,643 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 21 msec 2023-02-13 05:00:36,657 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-02-13 05:00:36,660 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-02-13 05:00:36,662 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1077): Master has completed initialization 2.518sec 2023-02-13 05:00:36,664 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-02-13 05:00:36,666 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-02-13 05:00:36,666 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-02-13 05:00:36,667 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45905,1676264432900-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-02-13 05:00:36,669 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45905,1676264432900-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-02-13 05:00:36,677 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1166): Balancer post startup initialization complete, took 0 seconds 2023-02-13 05:00:36,716 DEBUG [Listener at localhost/42165] zookeeper.ReadOnlyZKClient(139): Connect 0x5dd484f7 to 127.0.0.1:63943 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 05:00:36,720 DEBUG [Listener at localhost/42165] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@18cf3751, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 05:00:36,734 DEBUG [hconnection-0x315dbdb4-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:36,745 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43856, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:36,754 INFO [Listener at localhost/42165] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,45905,1676264432900 2023-02-13 05:00:36,783 INFO [Listener at localhost/42165] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=360, OpenFileDescriptor=697, MaxFileDescriptor=60000, SystemLoadAverage=178, ProcessCount=168, AvailableMemoryMB=3666 2023-02-13 05:00:36,798 DEBUG [Listener at localhost/42165] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-02-13 05:00:36,801 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59360, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-02-13 05:00:36,813 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'testRestoreSnapshot-1676264436786', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-13 05:00:36,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:36,817 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676264436786 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 05:00:36,819 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676264436786 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 05:00:36,821 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testRestoreSnapshot-1676264436786" procId is: 9 2023-02-13 05:00:36,827 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:36,828 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:36,828 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:36,828 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:36,828 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:36,829 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 empty. 2023-02-13 05:00:36,829 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:36,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 05:00:36,830 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb empty. 2023-02-13 05:00:36,830 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 empty. 2023-02-13 05:00:36,830 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 empty. 2023-02-13 05:00:36,830 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 empty. 2023-02-13 05:00:36,832 DEBUG [HFileArchiver-3] backup.HFileArchiver(153): Directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab empty. 2023-02-13 05:00:36,832 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:36,832 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:36,832 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:36,832 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:36,833 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:36,833 DEBUG [HFileArchiver-3] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:36,833 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676264436786 regions 2023-02-13 05:00:36,854 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/.tabledesc/.tableinfo.0000000001 2023-02-13 05:00:36,856 INFO [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-0] regionserver.HRegion(7671): creating {ENCODED => d429d785f30801586e3db3ae3a9428b8, NAME => 'testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676264436786', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp 2023-02-13 05:00:36,861 INFO [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-1] regionserver.HRegion(7671): creating {ENCODED => d063b19c8c76cc04b197243b846f83ab, NAME => 'testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676264436786', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp 2023-02-13 05:00:36,861 INFO [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-2] regionserver.HRegion(7671): creating {ENCODED => 11ed57e84fcc1f7d47e9ad2ed09021f8, NAME => 'testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676264436786', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp 2023-02-13 05:00:36,927 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:36,931 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-0] regionserver.HRegion(1603): Closing d429d785f30801586e3db3ae3a9428b8, disabling compactions & flushes 2023-02-13 05:00:36,931 INFO [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:36,932 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:36,932 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. after waiting 0 ms 2023-02-13 05:00:36,932 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:36,932 INFO [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:36,933 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-0] regionserver.HRegion(1557): Region close journal for d429d785f30801586e3db3ae3a9428b8: Waiting for close lock at 1676264436931Disabling compacts and flushes for region at 1676264436931Disabling writes for close at 1676264436932 (+1 ms)Writing region close event to WAL at 1676264436932Closed at 1676264436932 2023-02-13 05:00:36,933 INFO [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-0] regionserver.HRegion(7671): creating {ENCODED => 347ea8719201fbf1ee1e731b16986087, NAME => 'testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676264436786', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp 2023-02-13 05:00:36,934 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:36,935 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-2] regionserver.HRegion(1603): Closing 11ed57e84fcc1f7d47e9ad2ed09021f8, disabling compactions & flushes 2023-02-13 05:00:36,935 INFO [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:36,935 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:36,935 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. after waiting 0 ms 2023-02-13 05:00:36,936 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:36,936 INFO [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:36,936 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-2] regionserver.HRegion(1557): Region close journal for 11ed57e84fcc1f7d47e9ad2ed09021f8: Waiting for close lock at 1676264436935Disabling compacts and flushes for region at 1676264436935Disabling writes for close at 1676264436936 (+1 ms)Writing region close event to WAL at 1676264436936Closed at 1676264436936 2023-02-13 05:00:36,936 INFO [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-2] regionserver.HRegion(7671): creating {ENCODED => ee2d32fb6918e03288e862be581d9bcb, NAME => 'testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676264436786', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp 2023-02-13 05:00:36,937 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:36,937 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-1] regionserver.HRegion(1603): Closing d063b19c8c76cc04b197243b846f83ab, disabling compactions & flushes 2023-02-13 05:00:36,938 INFO [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:36,938 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:36,938 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. after waiting 0 ms 2023-02-13 05:00:36,938 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:36,938 INFO [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:36,938 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-1] regionserver.HRegion(1557): Region close journal for d063b19c8c76cc04b197243b846f83ab: Waiting for close lock at 1676264436937Disabling compacts and flushes for region at 1676264436937Disabling writes for close at 1676264436938 (+1 ms)Writing region close event to WAL at 1676264436938Closed at 1676264436938 2023-02-13 05:00:36,938 INFO [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-1] regionserver.HRegion(7671): creating {ENCODED => 7017e1f9205c2ebde9f23df4733114d5, NAME => 'testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676264436786', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp 2023-02-13 05:00:36,959 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:36,960 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-0] regionserver.HRegion(1603): Closing 347ea8719201fbf1ee1e731b16986087, disabling compactions & flushes 2023-02-13 05:00:36,960 INFO [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:36,960 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:36,960 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. after waiting 0 ms 2023-02-13 05:00:36,960 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:36,960 INFO [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:36,960 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-0] regionserver.HRegion(1557): Region close journal for 347ea8719201fbf1ee1e731b16986087: Waiting for close lock at 1676264436960Disabling compacts and flushes for region at 1676264436960Disabling writes for close at 1676264436960Writing region close event to WAL at 1676264436960Closed at 1676264436960 2023-02-13 05:00:36,961 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:36,963 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-2] regionserver.HRegion(1603): Closing ee2d32fb6918e03288e862be581d9bcb, disabling compactions & flushes 2023-02-13 05:00:36,963 INFO [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:36,963 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:36,963 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. after waiting 0 ms 2023-02-13 05:00:36,963 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:36,963 INFO [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:36,963 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-2] regionserver.HRegion(1557): Region close journal for ee2d32fb6918e03288e862be581d9bcb: Waiting for close lock at 1676264436963Disabling compacts and flushes for region at 1676264436963Disabling writes for close at 1676264436963Writing region close event to WAL at 1676264436963Closed at 1676264436963 2023-02-13 05:00:36,964 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:36,964 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-1] regionserver.HRegion(1603): Closing 7017e1f9205c2ebde9f23df4733114d5, disabling compactions & flushes 2023-02-13 05:00:36,965 INFO [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:36,965 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:36,965 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. after waiting 0 ms 2023-02-13 05:00:36,965 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:36,965 INFO [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:36,965 DEBUG [RegionOpenAndInit-testRestoreSnapshot-1676264436786-pool-1] regionserver.HRegion(1557): Region close journal for 7017e1f9205c2ebde9f23df4733114d5: Waiting for close lock at 1676264436964Disabling compacts and flushes for region at 1676264436964Disabling writes for close at 1676264436965 (+1 ms)Writing region close event to WAL at 1676264436965Closed at 1676264436965 2023-02-13 05:00:36,969 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676264436786 execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 05:00:36,971 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264436970"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264436970"}]},"ts":"1676264436970"} 2023-02-13 05:00:36,971 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264436970"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264436970"}]},"ts":"1676264436970"} 2023-02-13 05:00:36,971 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264436970"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264436970"}]},"ts":"1676264436970"} 2023-02-13 05:00:36,971 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264436970"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264436970"}]},"ts":"1676264436970"} 2023-02-13 05:00:36,972 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264436970"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264436970"}]},"ts":"1676264436970"} 2023-02-13 05:00:36,972 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264436970"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264436970"}]},"ts":"1676264436970"} 2023-02-13 05:00:37,016 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 05:00:37,018 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676264436786 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 05:00:37,018 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264437018"}]},"ts":"1676264437018"} 2023-02-13 05:00:37,020 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=ENABLING in hbase:meta 2023-02-13 05:00:37,024 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 05:00:37,029 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 05:00:37,029 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 05:00:37,029 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 05:00:37,029 DEBUG [PEWorker-2] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 05:00:37,029 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN}, {pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN}, {pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN}, {pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN}, {pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN}, {pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN}] 2023-02-13 05:00:37,032 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN 2023-02-13 05:00:37,032 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN 2023-02-13 05:00:37,033 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN 2023-02-13 05:00:37,033 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN 2023-02-13 05:00:37,034 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=11, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34363,1676264434089; forceNewPlan=false, retain=false 2023-02-13 05:00:37,034 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39885,1676264434059; forceNewPlan=false, retain=false 2023-02-13 05:00:37,034 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=12, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42589,1676264434015; forceNewPlan=false, retain=false 2023-02-13 05:00:37,034 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=13, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39885,1676264434059; forceNewPlan=false, retain=false 2023-02-13 05:00:37,035 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN 2023-02-13 05:00:37,036 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN 2023-02-13 05:00:37,036 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=15, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34363,1676264434089; forceNewPlan=false, retain=false 2023-02-13 05:00:37,037 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=14, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42589,1676264434015; forceNewPlan=false, retain=false 2023-02-13 05:00:37,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 05:00:37,184 INFO [jenkins-hbase4:45905] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 05:00:37,187 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:37,187 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:37,187 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:37,187 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264437187"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264437187"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264437187"}]},"ts":"1676264437187"} 2023-02-13 05:00:37,187 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:37,187 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264437187"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264437187"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264437187"}]},"ts":"1676264437187"} 2023-02-13 05:00:37,187 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:37,188 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264437187"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264437187"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264437187"}]},"ts":"1676264437187"} 2023-02-13 05:00:37,187 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264437187"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264437187"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264437187"}]},"ts":"1676264437187"} 2023-02-13 05:00:37,188 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264437187"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264437187"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264437187"}]},"ts":"1676264437187"} 2023-02-13 05:00:37,190 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=16, ppid=14, state=RUNNABLE; OpenRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:37,192 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=10, state=RUNNABLE; OpenRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:37,194 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=13, state=RUNNABLE; OpenRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:37,197 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=19, ppid=15, state=RUNNABLE; OpenRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:37,198 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:37,198 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264437198"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264437198"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264437198"}]},"ts":"1676264437198"} 2023-02-13 05:00:37,199 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=12, state=RUNNABLE; OpenRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:37,204 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=11, state=RUNNABLE; OpenRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:37,344 DEBUG [RSProcedureDispatcher-pool-2] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:37,345 DEBUG [RSProcedureDispatcher-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-02-13 05:00:37,348 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41616, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-02-13 05:00:37,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:37,355 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:37,355 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 347ea8719201fbf1ee1e731b16986087, NAME => 'testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 05:00:37,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ee2d32fb6918e03288e862be581d9bcb, NAME => 'testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 05:00:37,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:37,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:37,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:37,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:37,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:37,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:37,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:37,356 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:37,363 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:37,363 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d063b19c8c76cc04b197243b846f83ab, NAME => 'testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 05:00:37,363 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:37,363 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:37,363 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:37,363 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:37,364 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:37,364 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:37,366 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf 2023-02-13 05:00:37,367 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf 2023-02-13 05:00:37,367 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:37,369 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf 2023-02-13 05:00:37,369 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf 2023-02-13 05:00:37,370 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee2d32fb6918e03288e862be581d9bcb columnFamilyName cf 2023-02-13 05:00:37,370 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d063b19c8c76cc04b197243b846f83ab columnFamilyName cf 2023-02-13 05:00:37,370 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf 2023-02-13 05:00:37,371 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf 2023-02-13 05:00:37,372 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 347ea8719201fbf1ee1e731b16986087 columnFamilyName cf 2023-02-13 05:00:37,373 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(310): Store=347ea8719201fbf1ee1e731b16986087/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:37,373 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(310): Store=ee2d32fb6918e03288e862be581d9bcb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:37,376 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:37,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:37,377 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(310): Store=d063b19c8c76cc04b197243b846f83ab/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:37,377 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:37,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:37,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:37,380 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:37,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:37,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:37,385 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:37,391 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 05:00:37,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 05:00:37,393 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ee2d32fb6918e03288e862be581d9bcb; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71384630, jitterRate=0.06371387839317322}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:37,393 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ee2d32fb6918e03288e862be581d9bcb: Running coprocessor pre-open hook at 1676264437356Writing region info on filesystem at 1676264437356Initializing all the Stores at 1676264437358 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264437358Cleaning up temporary data from old regions at 1676264437379 (+21 ms)Cleaning up detritus from prior splits at 1676264437380 (+1 ms)Running coprocessor post-open hooks at 1676264437393 (+13 ms)Region opened successfully at 1676264437393 2023-02-13 05:00:37,394 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 05:00:37,394 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 347ea8719201fbf1ee1e731b16986087; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71554466, jitterRate=0.06624463200569153}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:37,395 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 347ea8719201fbf1ee1e731b16986087: Running coprocessor pre-open hook at 1676264437357Writing region info on filesystem at 1676264437357Initializing all the Stores at 1676264437358 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264437358Cleaning up temporary data from old regions at 1676264437379 (+21 ms)Cleaning up detritus from prior splits at 1676264437380 (+1 ms)Running coprocessor post-open hooks at 1676264437395 (+15 ms)Region opened successfully at 1676264437395 2023-02-13 05:00:37,395 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb., pid=16, masterSystemTime=1676264437344 2023-02-13 05:00:37,396 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087., pid=18, masterSystemTime=1676264437347 2023-02-13 05:00:37,399 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d063b19c8c76cc04b197243b846f83ab; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69416009, jitterRate=0.03437913954257965}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:37,399 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d063b19c8c76cc04b197243b846f83ab: Running coprocessor pre-open hook at 1676264437364Writing region info on filesystem at 1676264437364Initializing all the Stores at 1676264437365 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264437365Cleaning up temporary data from old regions at 1676264437381 (+16 ms)Cleaning up detritus from prior splits at 1676264437382 (+1 ms)Running coprocessor post-open hooks at 1676264437399 (+17 ms)Region opened successfully at 1676264437399 2023-02-13 05:00:37,401 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab., pid=21, masterSystemTime=1676264437352 2023-02-13 05:00:37,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:37,402 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:37,402 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:37,403 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 11ed57e84fcc1f7d47e9ad2ed09021f8, NAME => 'testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 05:00:37,403 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=14 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:37,403 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:37,404 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:37,404 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:37,404 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:37,404 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:37,404 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264437403"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264437403"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264437403"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264437403"}]},"ts":"1676264437403"} 2023-02-13 05:00:37,404 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:37,404 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:37,405 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d429d785f30801586e3db3ae3a9428b8, NAME => 'testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 05:00:37,405 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:37,405 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:37,405 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:37,405 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:37,405 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:37,405 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264437405"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264437405"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264437405"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264437405"}]},"ts":"1676264437405"} 2023-02-13 05:00:37,406 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:37,405 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:37,407 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:37,407 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7017e1f9205c2ebde9f23df4733114d5, NAME => 'testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 05:00:37,408 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=11 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:37,408 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264437407"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264437407"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264437407"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264437407"}]},"ts":"1676264437407"} 2023-02-13 05:00:37,408 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:37,409 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:37,409 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:37,409 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:37,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:37,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:37,412 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf 2023-02-13 05:00:37,412 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf 2023-02-13 05:00:37,412 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf 2023-02-13 05:00:37,412 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf 2023-02-13 05:00:37,412 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:37,412 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 11ed57e84fcc1f7d47e9ad2ed09021f8 columnFamilyName cf 2023-02-13 05:00:37,413 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d429d785f30801586e3db3ae3a9428b8 columnFamilyName cf 2023-02-13 05:00:37,414 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(310): Store=11ed57e84fcc1f7d47e9ad2ed09021f8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:37,416 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(310): Store=d429d785f30801586e3db3ae3a9428b8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:37,415 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf 2023-02-13 05:00:37,417 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf 2023-02-13 05:00:37,418 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7017e1f9205c2ebde9f23df4733114d5 columnFamilyName cf 2023-02-13 05:00:37,419 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:37,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:37,421 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=16, resume processing ppid=14 2023-02-13 05:00:37,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:37,422 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, ppid=14, state=SUCCESS; OpenRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,42589,1676264434015 in 222 msec 2023-02-13 05:00:37,422 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(310): Store=7017e1f9205c2ebde9f23df4733114d5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:37,422 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:37,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:37,425 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:37,426 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN in 392 msec 2023-02-13 05:00:37,426 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=13 2023-02-13 05:00:37,426 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=13, state=SUCCESS; OpenRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,39885,1676264434059 in 221 msec 2023-02-13 05:00:37,427 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=11 2023-02-13 05:00:37,427 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=11, state=SUCCESS; OpenRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,34363,1676264434089 in 215 msec 2023-02-13 05:00:37,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:37,431 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN in 397 msec 2023-02-13 05:00:37,434 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN in 398 msec 2023-02-13 05:00:37,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:37,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:37,436 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 05:00:37,437 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 11ed57e84fcc1f7d47e9ad2ed09021f8; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73044960, jitterRate=0.0884547233581543}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:37,437 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 11ed57e84fcc1f7d47e9ad2ed09021f8: Running coprocessor pre-open hook at 1676264437404Writing region info on filesystem at 1676264437404Initializing all the Stores at 1676264437408 (+4 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264437408Cleaning up temporary data from old regions at 1676264437422 (+14 ms)Cleaning up detritus from prior splits at 1676264437423 (+1 ms)Running coprocessor post-open hooks at 1676264437437 (+14 ms)Region opened successfully at 1676264437437 2023-02-13 05:00:37,438 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8., pid=20, masterSystemTime=1676264437344 2023-02-13 05:00:37,440 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:37,440 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:37,441 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=12 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:37,444 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 05:00:37,445 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264437441"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264437441"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264437441"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264437441"}]},"ts":"1676264437441"} 2023-02-13 05:00:37,445 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 05:00:37,445 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d429d785f30801586e3db3ae3a9428b8; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72799751, jitterRate=0.0848008245229721}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:37,446 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d429d785f30801586e3db3ae3a9428b8: Running coprocessor pre-open hook at 1676264437406Writing region info on filesystem at 1676264437406Initializing all the Stores at 1676264437408 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264437408Cleaning up temporary data from old regions at 1676264437423 (+15 ms)Cleaning up detritus from prior splits at 1676264437424 (+1 ms)Running coprocessor post-open hooks at 1676264437446 (+22 ms)Region opened successfully at 1676264437446 2023-02-13 05:00:37,447 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7017e1f9205c2ebde9f23df4733114d5; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73400049, jitterRate=0.09374596178531647}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:37,447 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7017e1f9205c2ebde9f23df4733114d5: Running coprocessor pre-open hook at 1676264437410Writing region info on filesystem at 1676264437410Initializing all the Stores at 1676264437412 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264437412Cleaning up temporary data from old regions at 1676264437427 (+15 ms)Cleaning up detritus from prior splits at 1676264437430 (+3 ms)Running coprocessor post-open hooks at 1676264437447 (+17 ms)Region opened successfully at 1676264437447 2023-02-13 05:00:37,447 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8., pid=17, masterSystemTime=1676264437347 2023-02-13 05:00:37,449 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5., pid=19, masterSystemTime=1676264437352 2023-02-13 05:00:37,451 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:37,451 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:37,453 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:37,453 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:37,453 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:37,453 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264437453"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264437453"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264437453"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264437453"}]},"ts":"1676264437453"} 2023-02-13 05:00:37,456 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=12 2023-02-13 05:00:37,456 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=15 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:37,457 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=12, state=SUCCESS; OpenRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,42589,1676264434015 in 250 msec 2023-02-13 05:00:37,457 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264437456"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264437456"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264437456"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264437456"}]},"ts":"1676264437456"} 2023-02-13 05:00:37,460 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=12, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN in 428 msec 2023-02-13 05:00:37,462 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=10 2023-02-13 05:00:37,463 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=10, state=SUCCESS; OpenRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059 in 265 msec 2023-02-13 05:00:37,465 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=19, resume processing ppid=15 2023-02-13 05:00:37,465 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=19, ppid=15, state=SUCCESS; OpenRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,34363,1676264434089 in 264 msec 2023-02-13 05:00:37,465 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN in 434 msec 2023-02-13 05:00:37,468 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=15, resume processing ppid=9 2023-02-13 05:00:37,468 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN in 436 msec 2023-02-13 05:00:37,469 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676264436786 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 05:00:37,469 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264437469"}]},"ts":"1676264437469"} 2023-02-13 05:00:37,471 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=ENABLED in hbase:meta 2023-02-13 05:00:37,475 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testRestoreSnapshot-1676264436786 execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 05:00:37,477 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=testRestoreSnapshot-1676264436786 in 662 msec 2023-02-13 05:00:37,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-02-13 05:00:37,588 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testRestoreSnapshot-1676264436786, procId: 9 completed 2023-02-13 05:00:37,594 DEBUG [Listener at localhost/42165] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676264436786 2023-02-13 05:00:37,595 DEBUG [Listener at localhost/42165] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:37,598 INFO [Listener at localhost/42165] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 05:00:37,621 DEBUG [hconnection-0x315dbdb4-shared-pool-5] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:37,624 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44910, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:37,625 DEBUG [hconnection-0x315dbdb4-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:37,629 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41628, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:37,641 INFO [Listener at localhost/42165] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676264436786 2023-02-13 05:00:37,645 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676264436786 2023-02-13 05:00:37,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=22, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:37,657 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264437657"}]},"ts":"1676264437657"} 2023-02-13 05:00:37,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-13 05:00:37,659 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=DISABLING in hbase:meta 2023-02-13 05:00:37,661 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676264436786 to state=DISABLING 2023-02-13 05:00:37,663 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN}, {pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN}, {pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN}, {pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN}, {pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN}, {pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN}] 2023-02-13 05:00:37,666 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN 2023-02-13 05:00:37,666 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=24, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN 2023-02-13 05:00:37,668 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=23, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN 2023-02-13 05:00:37,668 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=26, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN 2023-02-13 05:00:37,669 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=28, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN 2023-02-13 05:00:37,670 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:37,670 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264437670"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264437670"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264437670"}]},"ts":"1676264437670"} 2023-02-13 05:00:37,670 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:37,670 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:37,670 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:37,671 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264437670"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264437670"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264437670"}]},"ts":"1676264437670"} 2023-02-13 05:00:37,671 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264437670"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264437670"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264437670"}]},"ts":"1676264437670"} 2023-02-13 05:00:37,671 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264437670"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264437670"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264437670"}]},"ts":"1676264437670"} 2023-02-13 05:00:37,671 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:37,671 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264437671"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264437671"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264437671"}]},"ts":"1676264437671"} 2023-02-13 05:00:37,673 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=25, state=RUNNABLE; CloseRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:37,674 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=24, state=RUNNABLE; CloseRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:37,675 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=31, ppid=23, state=RUNNABLE; CloseRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:37,676 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=26, state=RUNNABLE; CloseRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:37,678 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=28, state=RUNNABLE; CloseRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:37,684 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=27, ppid=22, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN 2023-02-13 05:00:37,686 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:37,686 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264437686"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264437686"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264437686"}]},"ts":"1676264437686"} 2023-02-13 05:00:37,689 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=34, ppid=27, state=RUNNABLE; CloseRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:37,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:37,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:37,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:37,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7017e1f9205c2ebde9f23df4733114d5, disabling compactions & flushes 2023-02-13 05:00:37,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d429d785f30801586e3db3ae3a9428b8, disabling compactions & flushes 2023-02-13 05:00:37,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:37,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:37,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:37,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:37,837 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ee2d32fb6918e03288e862be581d9bcb, disabling compactions & flushes 2023-02-13 05:00:37,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. after waiting 0 ms 2023-02-13 05:00:37,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. after waiting 0 ms 2023-02-13 05:00:37,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:37,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:37,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:37,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:37,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. after waiting 0 ms 2023-02-13 05:00:37,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:37,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 05:00:37,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 05:00:37,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 05:00:37,847 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:37,847 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:37,847 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7017e1f9205c2ebde9f23df4733114d5: Waiting for close lock at 1676264437837Running coprocessor pre-close hooks at 1676264437837Disabling compacts and flushes for region at 1676264437837Disabling writes for close at 1676264437838 (+1 ms)Writing region close event to WAL at 1676264437839 (+1 ms)Running coprocessor post-close hooks at 1676264437847 (+8 ms)Closed at 1676264437847 2023-02-13 05:00:37,847 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d429d785f30801586e3db3ae3a9428b8: Waiting for close lock at 1676264437838Running coprocessor pre-close hooks at 1676264437838Disabling compacts and flushes for region at 1676264437838Disabling writes for close at 1676264437838Writing region close event to WAL at 1676264437838Running coprocessor post-close hooks at 1676264437847 (+9 ms)Closed at 1676264437847 2023-02-13 05:00:37,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:37,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ee2d32fb6918e03288e862be581d9bcb: Waiting for close lock at 1676264437837Running coprocessor pre-close hooks at 1676264437837Disabling compacts and flushes for region at 1676264437837Disabling writes for close at 1676264437838 (+1 ms)Writing region close event to WAL at 1676264437840 (+2 ms)Running coprocessor post-close hooks at 1676264437847 (+7 ms)Closed at 1676264437848 (+1 ms) 2023-02-13 05:00:37,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:37,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:37,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d063b19c8c76cc04b197243b846f83ab, disabling compactions & flushes 2023-02-13 05:00:37,851 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:37,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:37,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. after waiting 0 ms 2023-02-13 05:00:37,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:37,853 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=28 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=CLOSED 2023-02-13 05:00:37,853 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264437853"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264437853"}]},"ts":"1676264437853"} 2023-02-13 05:00:37,855 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=23 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=CLOSED 2023-02-13 05:00:37,856 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264437855"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264437855"}]},"ts":"1676264437855"} 2023-02-13 05:00:37,856 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:37,856 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:37,856 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:37,856 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:37,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 347ea8719201fbf1ee1e731b16986087, disabling compactions & flushes 2023-02-13 05:00:37,857 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:37,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 11ed57e84fcc1f7d47e9ad2ed09021f8, disabling compactions & flushes 2023-02-13 05:00:37,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:37,857 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:37,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. after waiting 0 ms 2023-02-13 05:00:37,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:37,858 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=27 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=CLOSED 2023-02-13 05:00:37,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:37,858 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. after waiting 0 ms 2023-02-13 05:00:37,858 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264437857"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264437857"}]},"ts":"1676264437857"} 2023-02-13 05:00:37,858 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:37,864 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=28 2023-02-13 05:00:37,864 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=28, state=SUCCESS; CloseRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,34363,1676264434089 in 178 msec 2023-02-13 05:00:37,865 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=31, resume processing ppid=23 2023-02-13 05:00:37,866 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=31, ppid=23, state=SUCCESS; CloseRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059 in 184 msec 2023-02-13 05:00:37,866 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=34, resume processing ppid=27 2023-02-13 05:00:37,866 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=34, ppid=27, state=SUCCESS; CloseRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,42589,1676264434015 in 174 msec 2023-02-13 05:00:37,866 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=28, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN in 201 msec 2023-02-13 05:00:37,867 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN in 203 msec 2023-02-13 05:00:37,869 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=27, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN in 204 msec 2023-02-13 05:00:37,871 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 05:00:37,872 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 05:00:37,872 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 05:00:37,872 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:37,872 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 347ea8719201fbf1ee1e731b16986087: Waiting for close lock at 1676264437857Running coprocessor pre-close hooks at 1676264437857Disabling compacts and flushes for region at 1676264437857Disabling writes for close at 1676264437857Writing region close event to WAL at 1676264437859 (+2 ms)Running coprocessor post-close hooks at 1676264437872 (+13 ms)Closed at 1676264437872 2023-02-13 05:00:37,873 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:37,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d063b19c8c76cc04b197243b846f83ab: Waiting for close lock at 1676264437850Running coprocessor pre-close hooks at 1676264437850Disabling compacts and flushes for region at 1676264437850Disabling writes for close at 1676264437851 (+1 ms)Writing region close event to WAL at 1676264437851Running coprocessor post-close hooks at 1676264437872 (+21 ms)Closed at 1676264437873 (+1 ms) 2023-02-13 05:00:37,873 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:37,873 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 11ed57e84fcc1f7d47e9ad2ed09021f8: Waiting for close lock at 1676264437857Running coprocessor pre-close hooks at 1676264437857Disabling compacts and flushes for region at 1676264437857Disabling writes for close at 1676264437858 (+1 ms)Writing region close event to WAL at 1676264437862 (+4 ms)Running coprocessor post-close hooks at 1676264437873 (+11 ms)Closed at 1676264437873 2023-02-13 05:00:37,874 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:37,875 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=26 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=CLOSED 2023-02-13 05:00:37,875 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264437875"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264437875"}]},"ts":"1676264437875"} 2023-02-13 05:00:37,876 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:37,877 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=24 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=CLOSED 2023-02-13 05:00:37,877 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264437877"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264437877"}]},"ts":"1676264437877"} 2023-02-13 05:00:37,877 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:37,878 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=CLOSED 2023-02-13 05:00:37,878 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264437878"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264437878"}]},"ts":"1676264437878"} 2023-02-13 05:00:37,881 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=26 2023-02-13 05:00:37,881 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=26, state=SUCCESS; CloseRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,39885,1676264434059 in 201 msec 2023-02-13 05:00:37,882 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=24 2023-02-13 05:00:37,882 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=24, state=SUCCESS; CloseRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,34363,1676264434089 in 206 msec 2023-02-13 05:00:37,883 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN in 218 msec 2023-02-13 05:00:37,884 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=25 2023-02-13 05:00:37,884 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=25, state=SUCCESS; CloseRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,42589,1676264434015 in 208 msec 2023-02-13 05:00:37,885 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=24, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN in 219 msec 2023-02-13 05:00:37,886 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=22 2023-02-13 05:00:37,887 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=22, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN in 221 msec 2023-02-13 05:00:37,888 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264437887"}]},"ts":"1676264437887"} 2023-02-13 05:00:37,889 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=DISABLED in hbase:meta 2023-02-13 05:00:37,891 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676264436786 to state=DISABLED 2023-02-13 05:00:37,894 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676264436786 in 245 msec 2023-02-13 05:00:37,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=22 2023-02-13 05:00:37,910 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676264436786, procId: 22 completed 2023-02-13 05:00:37,924 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } 2023-02-13 05:00:37,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676264437924 (current time:1676264437924). 2023-02-13 05:00:37,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 05:00:37,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] zookeeper.ReadOnlyZKClient(139): Connect 0x34fd747e to 127.0.0.1:63943 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 05:00:37,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@58642540, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 05:00:37,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:37,939 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43862, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:37,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34fd747e to 127.0.0.1:63943 2023-02-13 05:00:37,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:00:37,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 05:00:37,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } 2023-02-13 05:00:37,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE 2023-02-13 05:00:37,966 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=35, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE 2023-02-13 05:00:37,968 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=35, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE; timeout=600000, timestamp=1676265037968 2023-02-13 05:00:37,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } 2023-02-13 05:00:37,968 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676264436786 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676264436786 2023-02-13 05:00:37,973 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 05:00:37,973 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 05:00:37,988 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676264436786 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } 2023-02-13 05:00:37,991 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:37,991 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:37,992 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:37,992 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:37,993 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:37,993 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:37,993 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:37,998 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:37,999 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:37,998 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:37,998 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:37,998 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:37,998 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:37,998 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:38,008 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:00:38,008 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:00:38,008 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:00:38,008 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:00:38,009 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:00:38,008 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:00:38,008 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:00:38,223 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 05:00:38,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } is done 2023-02-13 05:00:38,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(404): Snapshoting '{ ss=emptySnaptb-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 }' is still in progress! 2023-02-13 05:00:38,227 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2428): (#2) Sleeping: 500ms while waiting for snapshot completion. 2023-02-13 05:00:38,510 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676264436786 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } as finished. 2023-02-13 05:00:38,510 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676264436786 2023-02-13 05:00:38,512 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/emptySnaptb-1676264436786 2023-02-13 05:00:38,552 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/emptySnaptb-1676264436786 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/emptySnaptb-1676264436786 2023-02-13 05:00:38,584 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676264436786 of table testRestoreSnapshot-1676264436786 completed 2023-02-13 05:00:38,584 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/emptySnaptb-1676264436786 2023-02-13 05:00:38,585 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/emptySnaptb-1676264436786 2023-02-13 05:00:38,585 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676264436786 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676264436786 at 1676264437968Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676264436786 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } at 1676264437988 (+20 ms)Consolidate snapshot: emptySnaptb-1676264436786 at 1676264438510 (+522 ms)Loading Region manifests for emptySnaptb-1676264436786 at 1676264438510Writing data manifest for emptySnaptb-1676264436786 at 1676264438524 (+14 ms)Verifying snapshot: emptySnaptb-1676264436786 at 1676264438540 (+16 ms)Snapshot emptySnaptb-1676264436786 of table testRestoreSnapshot-1676264436786 completed at 1676264438584 (+44 ms) 2023-02-13 05:00:38,586 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=35, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE 2023-02-13 05:00:38,588 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE in 629 msec 2023-02-13 05:00:38,728 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 05:00:38,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } is done 2023-02-13 05:00:38,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 05:00:38,731 INFO [Listener at localhost/42165] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676264436786 2023-02-13 05:00:38,735 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676264436786 2023-02-13 05:00:38,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=36, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:38,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-13 05:00:38,746 INFO [PEWorker-4] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676264436786 2023-02-13 05:00:38,746 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264438746"}]},"ts":"1676264438746"} 2023-02-13 05:00:38,748 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=ENABLING in hbase:meta 2023-02-13 05:00:38,751 DEBUG [PEWorker-4] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 05:00:38,754 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN}, {pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN}, {pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN}, {pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN}, {pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN}, {pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN}] 2023-02-13 05:00:38,756 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN 2023-02-13 05:00:38,757 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN 2023-02-13 05:00:38,757 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN 2023-02-13 05:00:38,757 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN 2023-02-13 05:00:38,758 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:38,758 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=38, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,34363,1676264434089; forceNewPlan=false, retain=true 2023-02-13 05:00:38,758 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:38,758 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=37, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39885,1676264434059; forceNewPlan=false, retain=true 2023-02-13 05:00:38,759 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:38,759 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=39, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42589,1676264434015; forceNewPlan=false, retain=true 2023-02-13 05:00:38,759 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:38,759 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=42, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,34363,1676264434089; forceNewPlan=false, retain=true 2023-02-13 05:00:38,759 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN 2023-02-13 05:00:38,760 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN 2023-02-13 05:00:38,761 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:38,761 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:38,761 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=41, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42589,1676264434015; forceNewPlan=false, retain=true 2023-02-13 05:00:38,761 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=36, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39885,1676264434059; forceNewPlan=false, retain=true 2023-02-13 05:00:38,908 INFO [jenkins-hbase4:45905] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 05:00:38,909 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:38,909 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:38,909 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:38,909 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264438909"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264438909"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264438909"}]},"ts":"1676264438909"} 2023-02-13 05:00:38,909 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:38,909 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:38,909 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264438909"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264438909"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264438909"}]},"ts":"1676264438909"} 2023-02-13 05:00:38,909 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264438909"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264438909"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264438909"}]},"ts":"1676264438909"} 2023-02-13 05:00:38,910 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264438909"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264438909"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264438909"}]},"ts":"1676264438909"} 2023-02-13 05:00:38,910 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264438909"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264438909"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264438909"}]},"ts":"1676264438909"} 2023-02-13 05:00:38,912 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=43, ppid=40, state=RUNNABLE; OpenRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:38,913 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=42, state=RUNNABLE; OpenRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:38,914 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=41, state=RUNNABLE; OpenRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:38,916 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=46, ppid=37, state=RUNNABLE; OpenRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:38,918 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:38,918 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264438918"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264438918"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264438918"}]},"ts":"1676264438918"} 2023-02-13 05:00:38,918 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=47, ppid=39, state=RUNNABLE; OpenRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:38,920 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=38, state=RUNNABLE; OpenRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:38,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-13 05:00:39,070 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:39,070 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:39,070 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 347ea8719201fbf1ee1e731b16986087, NAME => 'testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 05:00:39,070 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7017e1f9205c2ebde9f23df4733114d5, NAME => 'testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 05:00:39,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:39,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:39,071 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:39,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:39,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ee2d32fb6918e03288e862be581d9bcb, NAME => 'testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 05:00:39,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:39,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:39,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:39,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:39,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:39,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:39,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:39,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:39,071 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:39,073 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:39,073 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:39,073 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:39,074 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf 2023-02-13 05:00:39,074 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf 2023-02-13 05:00:39,075 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf 2023-02-13 05:00:39,075 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf 2023-02-13 05:00:39,075 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7017e1f9205c2ebde9f23df4733114d5 columnFamilyName cf 2023-02-13 05:00:39,075 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf 2023-02-13 05:00:39,075 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf 2023-02-13 05:00:39,076 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee2d32fb6918e03288e862be581d9bcb columnFamilyName cf 2023-02-13 05:00:39,076 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 347ea8719201fbf1ee1e731b16986087 columnFamilyName cf 2023-02-13 05:00:39,076 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(310): Store=7017e1f9205c2ebde9f23df4733114d5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:39,077 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(310): Store=347ea8719201fbf1ee1e731b16986087/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:39,077 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:39,077 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(310): Store=ee2d32fb6918e03288e862be581d9bcb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:39,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:39,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:39,079 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:39,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:39,081 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:39,083 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:39,084 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7017e1f9205c2ebde9f23df4733114d5; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71035001, jitterRate=0.058504000306129456}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:39,084 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7017e1f9205c2ebde9f23df4733114d5: Running coprocessor pre-open hook at 1676264439071Writing region info on filesystem at 1676264439071Initializing all the Stores at 1676264439072 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264439073 (+1 ms)Cleaning up temporary data from old regions at 1676264439080 (+7 ms)Cleaning up detritus from prior splits at 1676264439080Running coprocessor post-open hooks at 1676264439084 (+4 ms)Region opened successfully at 1676264439084 2023-02-13 05:00:39,085 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:39,085 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:39,086 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5., pid=44, masterSystemTime=1676264439065 2023-02-13 05:00:39,087 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ee2d32fb6918e03288e862be581d9bcb; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69856263, jitterRate=0.04093943536281586}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:39,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ee2d32fb6918e03288e862be581d9bcb: Running coprocessor pre-open hook at 1676264439071Writing region info on filesystem at 1676264439071Initializing all the Stores at 1676264439072 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264439073 (+1 ms)Cleaning up temporary data from old regions at 1676264439081 (+8 ms)Cleaning up detritus from prior splits at 1676264439082 (+1 ms)Running coprocessor post-open hooks at 1676264439087 (+5 ms)Region opened successfully at 1676264439087 2023-02-13 05:00:39,087 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 347ea8719201fbf1ee1e731b16986087; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72852766, jitterRate=0.08559080958366394}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:39,087 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 347ea8719201fbf1ee1e731b16986087: Running coprocessor pre-open hook at 1676264439071Writing region info on filesystem at 1676264439071Initializing all the Stores at 1676264439072 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264439073 (+1 ms)Cleaning up temporary data from old regions at 1676264439081 (+8 ms)Cleaning up detritus from prior splits at 1676264439082 (+1 ms)Running coprocessor post-open hooks at 1676264439087 (+5 ms)Region opened successfully at 1676264439087 2023-02-13 05:00:39,088 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb., pid=45, masterSystemTime=1676264439067 2023-02-13 05:00:39,088 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087., pid=43, masterSystemTime=1676264439065 2023-02-13 05:00:39,089 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:39,090 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:39,090 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:39,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d063b19c8c76cc04b197243b846f83ab, NAME => 'testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 05:00:39,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:39,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:39,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:39,090 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:39,091 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=42 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:39,091 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264439091"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264439091"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264439091"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264439091"}]},"ts":"1676264439091"} 2023-02-13 05:00:39,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:39,092 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:39,092 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:39,092 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 11ed57e84fcc1f7d47e9ad2ed09021f8, NAME => 'testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 05:00:39,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:39,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:39,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:39,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:39,093 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=41 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:39,093 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264439093"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264439093"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264439093"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264439093"}]},"ts":"1676264439093"} 2023-02-13 05:00:39,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:39,093 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:39,093 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:39,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d429d785f30801586e3db3ae3a9428b8, NAME => 'testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 05:00:39,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:39,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:39,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:39,094 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:39,095 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:39,095 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:39,095 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:39,095 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264439095"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264439095"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264439095"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264439095"}]},"ts":"1676264439095"} 2023-02-13 05:00:39,097 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf 2023-02-13 05:00:39,097 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf 2023-02-13 05:00:39,097 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf 2023-02-13 05:00:39,097 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf 2023-02-13 05:00:39,097 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:39,098 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 11ed57e84fcc1f7d47e9ad2ed09021f8 columnFamilyName cf 2023-02-13 05:00:39,098 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d063b19c8c76cc04b197243b846f83ab columnFamilyName cf 2023-02-13 05:00:39,101 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf 2023-02-13 05:00:39,101 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf 2023-02-13 05:00:39,101 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d429d785f30801586e3db3ae3a9428b8 columnFamilyName cf 2023-02-13 05:00:39,101 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(310): Store=d063b19c8c76cc04b197243b846f83ab/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:39,102 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(310): Store=11ed57e84fcc1f7d47e9ad2ed09021f8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:39,102 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=42 2023-02-13 05:00:39,102 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=42, state=SUCCESS; OpenRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,34363,1676264434089 in 182 msec 2023-02-13 05:00:39,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:39,104 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:39,104 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=41 2023-02-13 05:00:39,104 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=41, state=SUCCESS; OpenRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,42589,1676264434015 in 184 msec 2023-02-13 05:00:39,105 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN in 350 msec 2023-02-13 05:00:39,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:39,106 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(310): Store=d429d785f30801586e3db3ae3a9428b8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:39,106 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=43, resume processing ppid=40 2023-02-13 05:00:39,106 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=43, ppid=40, state=SUCCESS; OpenRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,39885,1676264434059 in 188 msec 2023-02-13 05:00:39,108 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:39,108 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN in 352 msec 2023-02-13 05:00:39,108 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:39,109 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN in 354 msec 2023-02-13 05:00:39,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:39,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:39,112 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:39,113 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 11ed57e84fcc1f7d47e9ad2ed09021f8; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61236780, jitterRate=-0.08750087022781372}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:39,113 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 11ed57e84fcc1f7d47e9ad2ed09021f8: Running coprocessor pre-open hook at 1676264439093Writing region info on filesystem at 1676264439093Initializing all the Stores at 1676264439094 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264439094Cleaning up temporary data from old regions at 1676264439107 (+13 ms)Cleaning up detritus from prior splits at 1676264439108 (+1 ms)Running coprocessor post-open hooks at 1676264439113 (+5 ms)Region opened successfully at 1676264439113 2023-02-13 05:00:39,114 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d063b19c8c76cc04b197243b846f83ab; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72751943, jitterRate=0.08408842980861664}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:39,114 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d063b19c8c76cc04b197243b846f83ab: Running coprocessor pre-open hook at 1676264439091Writing region info on filesystem at 1676264439091Initializing all the Stores at 1676264439092 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264439092Cleaning up temporary data from old regions at 1676264439109 (+17 ms)Cleaning up detritus from prior splits at 1676264439109Running coprocessor post-open hooks at 1676264439114 (+5 ms)Region opened successfully at 1676264439114 2023-02-13 05:00:39,114 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8., pid=47, masterSystemTime=1676264439067 2023-02-13 05:00:39,115 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab., pid=48, masterSystemTime=1676264439065 2023-02-13 05:00:39,116 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:39,117 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:39,117 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:39,117 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d429d785f30801586e3db3ae3a9428b8; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71833789, jitterRate=0.07040686905384064}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:39,117 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d429d785f30801586e3db3ae3a9428b8: Running coprocessor pre-open hook at 1676264439095Writing region info on filesystem at 1676264439095Initializing all the Stores at 1676264439097 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264439097Cleaning up temporary data from old regions at 1676264439111 (+14 ms)Cleaning up detritus from prior splits at 1676264439111Running coprocessor post-open hooks at 1676264439117 (+6 ms)Region opened successfully at 1676264439117 2023-02-13 05:00:39,118 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=39 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:39,118 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264439118"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264439118"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264439118"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264439118"}]},"ts":"1676264439118"} 2023-02-13 05:00:39,118 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8., pid=46, masterSystemTime=1676264439065 2023-02-13 05:00:39,119 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:39,119 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:39,121 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=38 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:39,121 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264439120"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264439120"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264439120"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264439120"}]},"ts":"1676264439120"} 2023-02-13 05:00:39,121 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:39,121 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:39,123 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=37 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:39,123 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264439123"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264439123"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264439123"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264439123"}]},"ts":"1676264439123"} 2023-02-13 05:00:39,127 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=47, resume processing ppid=39 2023-02-13 05:00:39,127 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=47, ppid=39, state=SUCCESS; OpenRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,42589,1676264434015 in 202 msec 2023-02-13 05:00:39,129 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=38 2023-02-13 05:00:39,129 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=38, state=SUCCESS; OpenRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,34363,1676264434089 in 205 msec 2023-02-13 05:00:39,130 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN in 375 msec 2023-02-13 05:00:39,130 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=46, resume processing ppid=37 2023-02-13 05:00:39,130 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=46, ppid=37, state=SUCCESS; OpenRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059 in 210 msec 2023-02-13 05:00:39,132 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN in 377 msec 2023-02-13 05:00:39,133 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-02-13 05:00:39,134 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN in 378 msec 2023-02-13 05:00:39,134 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264439134"}]},"ts":"1676264439134"} 2023-02-13 05:00:39,136 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=ENABLED in hbase:meta 2023-02-13 05:00:39,138 INFO [PEWorker-3] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676264436786' was successfully enabled. 2023-02-13 05:00:39,141 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=36, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676264436786 in 402 msec 2023-02-13 05:00:39,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=36 2023-02-13 05:00:39,500 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676264436786, procId: 36 completed 2023-02-13 05:00:39,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:00:39,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:00:39,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:00:39,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:00:39,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:00:39,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:00:39,558 DEBUG [Listener at localhost/42165] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676264436786 2023-02-13 05:00:39,559 DEBUG [Listener at localhost/42165] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:39,560 INFO [Listener at localhost/42165] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 05:00:39,581 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 05:00:39,583 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 05:00:39,585 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 05:00:39,587 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 05:00:39,589 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 05:00:39,598 INFO [Listener at localhost/42165] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676264436786 2023-02-13 05:00:39,598 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676264436786 2023-02-13 05:00:39,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=49, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:39,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-13 05:00:39,604 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264439604"}]},"ts":"1676264439604"} 2023-02-13 05:00:39,606 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=DISABLING in hbase:meta 2023-02-13 05:00:39,608 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676264436786 to state=DISABLING 2023-02-13 05:00:39,609 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN}, {pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN}, {pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN}, {pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN}, {pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN}, {pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN}] 2023-02-13 05:00:39,611 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=52, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN 2023-02-13 05:00:39,611 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=53, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN 2023-02-13 05:00:39,611 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=54, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN 2023-02-13 05:00:39,612 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN 2023-02-13 05:00:39,612 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN 2023-02-13 05:00:39,613 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:39,613 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:39,613 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:39,613 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264439613"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264439613"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264439613"}]},"ts":"1676264439613"} 2023-02-13 05:00:39,613 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264439613"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264439613"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264439613"}]},"ts":"1676264439613"} 2023-02-13 05:00:39,613 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:39,613 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:39,613 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264439613"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264439613"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264439613"}]},"ts":"1676264439613"} 2023-02-13 05:00:39,613 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264439613"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264439613"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264439613"}]},"ts":"1676264439613"} 2023-02-13 05:00:39,613 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264439613"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264439613"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264439613"}]},"ts":"1676264439613"} 2023-02-13 05:00:39,615 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=51, state=RUNNABLE; CloseRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:39,616 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=57, ppid=52, state=RUNNABLE; CloseRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:39,618 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=55, state=RUNNABLE; CloseRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:39,620 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=54, state=RUNNABLE; CloseRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:39,621 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=50, ppid=49, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN 2023-02-13 05:00:39,622 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=53, state=RUNNABLE; CloseRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:39,625 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:39,625 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264439625"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264439625"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264439625"}]},"ts":"1676264439625"} 2023-02-13 05:00:39,627 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=50, state=RUNNABLE; CloseRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:39,768 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:39,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d063b19c8c76cc04b197243b846f83ab, disabling compactions & flushes 2023-02-13 05:00:39,769 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:39,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:39,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. after waiting 0 ms 2023-02-13 05:00:39,769 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:39,770 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing d063b19c8c76cc04b197243b846f83ab 1/1 column families, dataSize=1.83 KB heapSize=4.19 KB 2023-02-13 05:00:39,770 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:39,771 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ee2d32fb6918e03288e862be581d9bcb, disabling compactions & flushes 2023-02-13 05:00:39,771 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:39,771 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:39,771 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. after waiting 0 ms 2023-02-13 05:00:39,771 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:39,771 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing ee2d32fb6918e03288e862be581d9bcb 1/1 column families, dataSize=1.96 KB heapSize=4.47 KB 2023-02-13 05:00:39,777 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:39,777 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d429d785f30801586e3db3ae3a9428b8, disabling compactions & flushes 2023-02-13 05:00:39,777 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:39,778 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:39,778 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. after waiting 0 ms 2023-02-13 05:00:39,778 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:39,778 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing d429d785f30801586e3db3ae3a9428b8 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-13 05:00:39,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-13 05:00:39,888 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 2023-02-13 05:00:39,888 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 2023-02-13 05:00:39,888 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 2023-02-13 05:00:39,922 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/.tmp/cf/811ac31223674808858dc07f9153be03 2023-02-13 05:00:39,922 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/.tmp/cf/5ba29d67350f40888be6bfe3fec7f441 2023-02-13 05:00:39,923 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/.tmp/cf/d3866b21c6944cf0882c65fec182cb40 2023-02-13 05:00:39,932 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/.tmp/cf/811ac31223674808858dc07f9153be03 as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/811ac31223674808858dc07f9153be03 2023-02-13 05:00:39,932 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/.tmp/cf/5ba29d67350f40888be6bfe3fec7f441 as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/5ba29d67350f40888be6bfe3fec7f441 2023-02-13 05:00:39,934 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/.tmp/cf/d3866b21c6944cf0882c65fec182cb40 as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/d3866b21c6944cf0882c65fec182cb40 2023-02-13 05:00:39,941 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/811ac31223674808858dc07f9153be03, entries=32, sequenceid=8, filesize=10.4 K 2023-02-13 05:00:39,941 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/5ba29d67350f40888be6bfe3fec7f441, entries=28, sequenceid=8, filesize=9.7 K 2023-02-13 05:00:39,942 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/d3866b21c6944cf0882c65fec182cb40, entries=30, sequenceid=8, filesize=10.0 K 2023-02-13 05:00:39,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for ee2d32fb6918e03288e862be581d9bcb in 174ms, sequenceid=8, compaction requested=false 2023-02-13 05:00:39,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for d063b19c8c76cc04b197243b846f83ab in 175ms, sequenceid=8, compaction requested=false 2023-02-13 05:00:39,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for d429d785f30801586e3db3ae3a9428b8 in 167ms, sequenceid=8, compaction requested=false 2023-02-13 05:00:39,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676264436786' 2023-02-13 05:00:39,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676264436786' 2023-02-13 05:00:39,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testRestoreSnapshot-1676264436786' 2023-02-13 05:00:39,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 05:00:39,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 05:00:39,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 05:00:39,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:39,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:39,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d429d785f30801586e3db3ae3a9428b8: Waiting for close lock at 1676264439777Running coprocessor pre-close hooks at 1676264439777Disabling compacts and flushes for region at 1676264439777Disabling writes for close at 1676264439778 (+1 ms)Obtaining lock to block concurrent updates at 1676264439778Preparing flush snapshotting stores in d429d785f30801586e3db3ae3a9428b8 at 1676264439778Finished memstore snapshotting testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8., syncing WAL and waiting on mvcc, flushsize=dataSize=2142, getHeapSize=4848, getOffHeapSize=0, getCellsCount=32 at 1676264439781 (+3 ms)Flushing stores of testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. at 1676264439781Flushing d429d785f30801586e3db3ae3a9428b8/cf: creating writer at 1676264439784 (+3 ms)Flushing mob file d429d785f30801586e3db3ae3a9428b8/cf: appending metadata at 1676264439820 (+36 ms)Flushing mob file d429d785f30801586e3db3ae3a9428b8/cf: closing flushed file at 1676264439820Flushing d429d785f30801586e3db3ae3a9428b8/cf: appending metadata at 1676264439890 (+70 ms)Flushing d429d785f30801586e3db3ae3a9428b8/cf: closing flushed file at 1676264439894 (+4 ms)Flushing d429d785f30801586e3db3ae3a9428b8/cf: reopening flushed file at 1676264439934 (+40 ms)Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for d429d785f30801586e3db3ae3a9428b8 in 167ms, sequenceid=8, compaction requested=false at 1676264439945 (+11 ms)Writing region close event to WAL at 1676264439954 (+9 ms)Running coprocessor post-close hooks at 1676264439959 (+5 ms)Closed at 1676264439959 2023-02-13 05:00:39,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:39,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ee2d32fb6918e03288e862be581d9bcb: Waiting for close lock at 1676264439770Running coprocessor pre-close hooks at 1676264439770Disabling compacts and flushes for region at 1676264439770Disabling writes for close at 1676264439771 (+1 ms)Obtaining lock to block concurrent updates at 1676264439771Preparing flush snapshotting stores in ee2d32fb6918e03288e862be581d9bcb at 1676264439771Finished memstore snapshotting testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb., syncing WAL and waiting on mvcc, flushsize=dataSize=2008, getHeapSize=4560, getOffHeapSize=0, getCellsCount=30 at 1676264439780 (+9 ms)Flushing stores of testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. at 1676264439781 (+1 ms)Flushing ee2d32fb6918e03288e862be581d9bcb/cf: creating writer at 1676264439784 (+3 ms)Flushing mob file ee2d32fb6918e03288e862be581d9bcb/cf: appending metadata at 1676264439819 (+35 ms)Flushing mob file ee2d32fb6918e03288e862be581d9bcb/cf: closing flushed file at 1676264439819Flushing ee2d32fb6918e03288e862be581d9bcb/cf: appending metadata at 1676264439890 (+71 ms)Flushing ee2d32fb6918e03288e862be581d9bcb/cf: closing flushed file at 1676264439894 (+4 ms)Flushing ee2d32fb6918e03288e862be581d9bcb/cf: reopening flushed file at 1676264439935 (+41 ms)Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for ee2d32fb6918e03288e862be581d9bcb in 174ms, sequenceid=8, compaction requested=false at 1676264439945 (+10 ms)Writing region close event to WAL at 1676264439954 (+9 ms)Running coprocessor post-close hooks at 1676264439959 (+5 ms)Closed at 1676264439959 2023-02-13 05:00:39,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d063b19c8c76cc04b197243b846f83ab: Waiting for close lock at 1676264439768Running coprocessor pre-close hooks at 1676264439768Disabling compacts and flushes for region at 1676264439768Disabling writes for close at 1676264439769 (+1 ms)Obtaining lock to block concurrent updates at 1676264439770 (+1 ms)Preparing flush snapshotting stores in d063b19c8c76cc04b197243b846f83ab at 1676264439770Finished memstore snapshotting testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab., syncing WAL and waiting on mvcc, flushsize=dataSize=1874, getHeapSize=4272, getOffHeapSize=0, getCellsCount=28 at 1676264439780 (+10 ms)Flushing stores of testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. at 1676264439781 (+1 ms)Flushing d063b19c8c76cc04b197243b846f83ab/cf: creating writer at 1676264439784 (+3 ms)Flushing mob file d063b19c8c76cc04b197243b846f83ab/cf: appending metadata at 1676264439820 (+36 ms)Flushing mob file d063b19c8c76cc04b197243b846f83ab/cf: closing flushed file at 1676264439820Flushing d063b19c8c76cc04b197243b846f83ab/cf: appending metadata at 1676264439890 (+70 ms)Flushing d063b19c8c76cc04b197243b846f83ab/cf: closing flushed file at 1676264439894 (+4 ms)Flushing d063b19c8c76cc04b197243b846f83ab/cf: reopening flushed file at 1676264439934 (+40 ms)Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for d063b19c8c76cc04b197243b846f83ab in 175ms, sequenceid=8, compaction requested=false at 1676264439945 (+11 ms)Writing region close event to WAL at 1676264439954 (+9 ms)Running coprocessor post-close hooks at 1676264439959 (+5 ms)Closed at 1676264439959 2023-02-13 05:00:39,961 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:39,961 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:39,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 347ea8719201fbf1ee1e731b16986087, disabling compactions & flushes 2023-02-13 05:00:39,962 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:39,962 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:39,962 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. after waiting 0 ms 2023-02-13 05:00:39,962 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:39,962 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 347ea8719201fbf1ee1e731b16986087 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-13 05:00:39,962 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=50 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=CLOSED 2023-02-13 05:00:39,962 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264439962"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264439962"}]},"ts":"1676264439962"} 2023-02-13 05:00:39,963 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:39,963 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:39,964 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 11ed57e84fcc1f7d47e9ad2ed09021f8, disabling compactions & flushes 2023-02-13 05:00:39,964 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:39,964 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:39,964 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. after waiting 0 ms 2023-02-13 05:00:39,964 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:39,964 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 11ed57e84fcc1f7d47e9ad2ed09021f8 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-02-13 05:00:39,966 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:39,966 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:39,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7017e1f9205c2ebde9f23df4733114d5, disabling compactions & flushes 2023-02-13 05:00:39,966 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:39,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:39,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. after waiting 0 ms 2023-02-13 05:00:39,967 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:39,967 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=54 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=CLOSED 2023-02-13 05:00:39,967 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 7017e1f9205c2ebde9f23df4733114d5 1/1 column families, dataSize=22.70 KB heapSize=49.05 KB 2023-02-13 05:00:39,967 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264439966"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264439966"}]},"ts":"1676264439966"} 2023-02-13 05:00:39,968 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=CLOSED 2023-02-13 05:00:39,969 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264439968"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264439968"}]},"ts":"1676264439968"} 2023-02-13 05:00:39,976 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=50 2023-02-13 05:00:39,976 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=50, state=SUCCESS; CloseRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059 in 340 msec 2023-02-13 05:00:39,978 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=54 2023-02-13 05:00:39,978 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=54, state=SUCCESS; CloseRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,42589,1676264434015 in 353 msec 2023-02-13 05:00:39,978 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=51 2023-02-13 05:00:39,979 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=51, state=SUCCESS; CloseRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,34363,1676264434089 in 360 msec 2023-02-13 05:00:39,979 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=50, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN in 367 msec 2023-02-13 05:00:39,980 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=54, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN in 369 msec 2023-02-13 05:00:39,981 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN in 370 msec 2023-02-13 05:00:40,035 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 2023-02-13 05:00:40,050 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=22.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/.tmp/cf/d15ef8effb63426f88cf28b863b418da 2023-02-13 05:00:40,057 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/.tmp/cf/d15ef8effb63426f88cf28b863b418da as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d15ef8effb63426f88cf28b863b418da 2023-02-13 05:00:40,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d15ef8effb63426f88cf28b863b418da, entries=347, sequenceid=8, filesize=64.0 K 2023-02-13 05:00:40,068 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.70 KB/23245, heapSize ~49.03 KB/50208, currentSize=0 B/0 for 7017e1f9205c2ebde9f23df4733114d5 in 100ms, sequenceid=8, compaction requested=false 2023-02-13 05:00:40,074 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 05:00:40,075 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:40,075 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7017e1f9205c2ebde9f23df4733114d5: Waiting for close lock at 1676264439966Running coprocessor pre-close hooks at 1676264439966Disabling compacts and flushes for region at 1676264439966Disabling writes for close at 1676264439967 (+1 ms)Obtaining lock to block concurrent updates at 1676264439967Preparing flush snapshotting stores in 7017e1f9205c2ebde9f23df4733114d5 at 1676264439967Finished memstore snapshotting testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5., syncing WAL and waiting on mvcc, flushsize=dataSize=23245, getHeapSize=50208, getOffHeapSize=0, getCellsCount=347 at 1676264439967Flushing stores of testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. at 1676264439969 (+2 ms)Flushing 7017e1f9205c2ebde9f23df4733114d5/cf: creating writer at 1676264439972 (+3 ms)Flushing mob file 7017e1f9205c2ebde9f23df4733114d5/cf: appending metadata at 1676264440013 (+41 ms)Flushing mob file 7017e1f9205c2ebde9f23df4733114d5/cf: closing flushed file at 1676264440013Flushing 7017e1f9205c2ebde9f23df4733114d5/cf: appending metadata at 1676264440036 (+23 ms)Flushing 7017e1f9205c2ebde9f23df4733114d5/cf: closing flushed file at 1676264440036Flushing 7017e1f9205c2ebde9f23df4733114d5/cf: reopening flushed file at 1676264440059 (+23 ms)Finished flush of dataSize ~22.70 KB/23245, heapSize ~49.03 KB/50208, currentSize=0 B/0 for 7017e1f9205c2ebde9f23df4733114d5 in 100ms, sequenceid=8, compaction requested=false at 1676264440068 (+9 ms)Writing region close event to WAL at 1676264440070 (+2 ms)Running coprocessor post-close hooks at 1676264440075 (+5 ms)Closed at 1676264440075 2023-02-13 05:00:40,077 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:40,078 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=CLOSED 2023-02-13 05:00:40,078 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264440078"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264440078"}]},"ts":"1676264440078"} 2023-02-13 05:00:40,082 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=55 2023-02-13 05:00:40,082 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=55, state=SUCCESS; CloseRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,34363,1676264434089 in 462 msec 2023-02-13 05:00:40,083 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN in 473 msec 2023-02-13 05:00:40,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-13 05:00:40,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 2023-02-13 05:00:40,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 2023-02-13 05:00:40,437 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/.tmp/cf/6414549fb99340299c9cb4b42e13bd55 2023-02-13 05:00:40,438 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/.tmp/cf/519efdc2aee8432c80befe61da0da81c 2023-02-13 05:00:40,446 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/.tmp/cf/519efdc2aee8432c80befe61da0da81c as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/519efdc2aee8432c80befe61da0da81c 2023-02-13 05:00:40,446 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/.tmp/cf/6414549fb99340299c9cb4b42e13bd55 as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/6414549fb99340299c9cb4b42e13bd55 2023-02-13 05:00:40,453 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/6414549fb99340299c9cb4b42e13bd55, entries=27, sequenceid=8, filesize=9.5 K 2023-02-13 05:00:40,453 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/519efdc2aee8432c80befe61da0da81c, entries=36, sequenceid=8, filesize=11.0 K 2023-02-13 05:00:40,456 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 347ea8719201fbf1ee1e731b16986087 in 494ms, sequenceid=8, compaction requested=false 2023-02-13 05:00:40,456 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 11ed57e84fcc1f7d47e9ad2ed09021f8 in 492ms, sequenceid=8, compaction requested=false 2023-02-13 05:00:40,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 05:00:40,465 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 05:00:40,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:40,465 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 11ed57e84fcc1f7d47e9ad2ed09021f8: Waiting for close lock at 1676264439963Running coprocessor pre-close hooks at 1676264439963Disabling compacts and flushes for region at 1676264439963Disabling writes for close at 1676264439964 (+1 ms)Obtaining lock to block concurrent updates at 1676264439964Preparing flush snapshotting stores in 11ed57e84fcc1f7d47e9ad2ed09021f8 at 1676264439964Finished memstore snapshotting testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8., syncing WAL and waiting on mvcc, flushsize=dataSize=1807, getHeapSize=4128, getOffHeapSize=0, getCellsCount=27 at 1676264439964Flushing stores of testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. at 1676264439968 (+4 ms)Flushing 11ed57e84fcc1f7d47e9ad2ed09021f8/cf: creating writer at 1676264439968Flushing mob file 11ed57e84fcc1f7d47e9ad2ed09021f8/cf: appending metadata at 1676264439981 (+13 ms)Flushing mob file 11ed57e84fcc1f7d47e9ad2ed09021f8/cf: closing flushed file at 1676264439981Flushing 11ed57e84fcc1f7d47e9ad2ed09021f8/cf: appending metadata at 1676264440421 (+440 ms)Flushing 11ed57e84fcc1f7d47e9ad2ed09021f8/cf: closing flushed file at 1676264440421Flushing 11ed57e84fcc1f7d47e9ad2ed09021f8/cf: reopening flushed file at 1676264440447 (+26 ms)Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for 11ed57e84fcc1f7d47e9ad2ed09021f8 in 492ms, sequenceid=8, compaction requested=false at 1676264440457 (+10 ms)Writing region close event to WAL at 1676264440460 (+3 ms)Running coprocessor post-close hooks at 1676264440465 (+5 ms)Closed at 1676264440465 2023-02-13 05:00:40,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:40,465 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 347ea8719201fbf1ee1e731b16986087: Waiting for close lock at 1676264439961Running coprocessor pre-close hooks at 1676264439961Disabling compacts and flushes for region at 1676264439961Disabling writes for close at 1676264439962 (+1 ms)Obtaining lock to block concurrent updates at 1676264439962Preparing flush snapshotting stores in 347ea8719201fbf1ee1e731b16986087 at 1676264439962Finished memstore snapshotting testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087., syncing WAL and waiting on mvcc, flushsize=dataSize=2410, getHeapSize=5424, getOffHeapSize=0, getCellsCount=36 at 1676264439962Flushing stores of testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. at 1676264439964 (+2 ms)Flushing 347ea8719201fbf1ee1e731b16986087/cf: creating writer at 1676264439965 (+1 ms)Flushing mob file 347ea8719201fbf1ee1e731b16986087/cf: appending metadata at 1676264439981 (+16 ms)Flushing mob file 347ea8719201fbf1ee1e731b16986087/cf: closing flushed file at 1676264439981Flushing 347ea8719201fbf1ee1e731b16986087/cf: appending metadata at 1676264440421 (+440 ms)Flushing 347ea8719201fbf1ee1e731b16986087/cf: closing flushed file at 1676264440421Flushing 347ea8719201fbf1ee1e731b16986087/cf: reopening flushed file at 1676264440446 (+25 ms)Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 347ea8719201fbf1ee1e731b16986087 in 494ms, sequenceid=8, compaction requested=false at 1676264440456 (+10 ms)Writing region close event to WAL at 1676264440458 (+2 ms)Running coprocessor post-close hooks at 1676264440465 (+7 ms)Closed at 1676264440465 2023-02-13 05:00:40,467 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:40,467 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=52 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=CLOSED 2023-02-13 05:00:40,468 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264440467"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264440467"}]},"ts":"1676264440467"} 2023-02-13 05:00:40,468 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:40,468 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=53 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=CLOSED 2023-02-13 05:00:40,469 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264440468"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264440468"}]},"ts":"1676264440468"} 2023-02-13 05:00:40,472 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=57, resume processing ppid=52 2023-02-13 05:00:40,472 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=57, ppid=52, state=SUCCESS; CloseRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,42589,1676264434015 in 853 msec 2023-02-13 05:00:40,473 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=53 2023-02-13 05:00:40,473 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=53, state=SUCCESS; CloseRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,39885,1676264434059 in 849 msec 2023-02-13 05:00:40,473 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN in 863 msec 2023-02-13 05:00:40,475 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=53, resume processing ppid=49 2023-02-13 05:00:40,475 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=53, ppid=49, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN in 864 msec 2023-02-13 05:00:40,475 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264440475"}]},"ts":"1676264440475"} 2023-02-13 05:00:40,477 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=DISABLED in hbase:meta 2023-02-13 05:00:40,479 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676264436786 to state=DISABLED 2023-02-13 05:00:40,481 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=49, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676264436786 in 881 msec 2023-02-13 05:00:41,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=49 2023-02-13 05:00:41,108 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676264436786, procId: 49 completed 2023-02-13 05:00:41,109 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } 2023-02-13 05:00:41,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676264441109 (current time:1676264441109). 2023-02-13 05:00:41,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 05:00:41,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] zookeeper.ReadOnlyZKClient(139): Connect 0x35b24570 to 127.0.0.1:63943 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 05:00:41,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4c42a729, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 05:00:41,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:41,118 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43874, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:41,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x35b24570 to 127.0.0.1:63943 2023-02-13 05:00:41,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:00:41,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 05:00:41,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } 2023-02-13 05:00:41,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE 2023-02-13 05:00:41,123 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=62, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE 2023-02-13 05:00:41,124 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=62, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE; timeout=600000, timestamp=1676265041124 2023-02-13 05:00:41,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } 2023-02-13 05:00:41,124 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676264436786 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676264436786 2023-02-13 05:00:41,125 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 05:00:41,125 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 05:00:41,141 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676264436786 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } 2023-02-13 05:00:41,143 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:41,143 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:41,143 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:41,143 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:41,143 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:41,143 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:41,143 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:41,144 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:41,144 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:41,144 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:41,144 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:41,144 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:41,148 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:41,148 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:41,151 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/6414549fb99340299c9cb4b42e13bd55] hfiles 2023-02-13 05:00:41,152 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/d3866b21c6944cf0882c65fec182cb40] hfiles 2023-02-13 05:00:41,152 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/6414549fb99340299c9cb4b42e13bd55 2023-02-13 05:00:41,152 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/d3866b21c6944cf0882c65fec182cb40 2023-02-13 05:00:41,152 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/811ac31223674808858dc07f9153be03] hfiles 2023-02-13 05:00:41,151 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/519efdc2aee8432c80befe61da0da81c] hfiles 2023-02-13 05:00:41,152 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/811ac31223674808858dc07f9153be03 2023-02-13 05:00:41,152 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/519efdc2aee8432c80befe61da0da81c 2023-02-13 05:00:41,152 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d15ef8effb63426f88cf28b863b418da] hfiles 2023-02-13 05:00:41,153 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d15ef8effb63426f88cf28b863b418da 2023-02-13 05:00:41,156 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2] hfiles 2023-02-13 05:00:41,156 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 2023-02-13 05:00:41,156 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 2023-02-13 05:00:41,156 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 2023-02-13 05:00:41,156 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 2023-02-13 05:00:41,156 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 2023-02-13 05:00:41,156 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 2023-02-13 05:00:41,156 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/5ba29d67350f40888be6bfe3fec7f441] hfiles 2023-02-13 05:00:41,157 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/5ba29d67350f40888be6bfe3fec7f441 2023-02-13 05:00:41,209 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676264436786 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } as finished. 2023-02-13 05:00:41,209 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676264436786 2023-02-13 05:00:41,210 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/snaptb0-1676264436786 2023-02-13 05:00:41,244 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/snaptb0-1676264436786 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/snaptb0-1676264436786 2023-02-13 05:00:41,247 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 05:00:41,304 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676264436786 of table testRestoreSnapshot-1676264436786 completed 2023-02-13 05:00:41,305 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/snaptb0-1676264436786 2023-02-13 05:00:41,305 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/snaptb0-1676264436786 2023-02-13 05:00:41,305 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676264436786 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676264436786 at 1676264441125Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676264436786 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } at 1676264441141 (+16 ms)Consolidate snapshot: snaptb0-1676264436786 at 1676264441209 (+68 ms)Loading Region manifests for snaptb0-1676264436786 at 1676264441209Writing data manifest for snaptb0-1676264436786 at 1676264441215 (+6 ms)Verifying snapshot: snaptb0-1676264436786 at 1676264441229 (+14 ms)Snapshot snaptb0-1676264436786 of table testRestoreSnapshot-1676264436786 completed at 1676264441304 (+75 ms) 2023-02-13 05:00:41,306 DEBUG [PEWorker-5] locking.LockProcedure(242): UNLOCKED pid=62, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE 2023-02-13 05:00:41,308 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE in 184 msec 2023-02-13 05:00:41,354 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-02-13 05:00:41,355 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-02-13 05:00:41,375 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 05:00:41,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } is done 2023-02-13 05:00:41,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 05:00:41,377 INFO [Listener at localhost/42165] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676264436786 2023-02-13 05:00:41,377 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676264436786 2023-02-13 05:00:41,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=63, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:41,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-13 05:00:41,382 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676264436786 2023-02-13 05:00:41,382 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264441382"}]},"ts":"1676264441382"} 2023-02-13 05:00:41,384 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=ENABLING in hbase:meta 2023-02-13 05:00:41,391 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 05:00:41,391 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN}, {pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN}, {pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN}, {pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN}, {pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN}, {pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN}] 2023-02-13 05:00:41,393 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN 2023-02-13 05:00:41,393 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN 2023-02-13 05:00:41,393 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN 2023-02-13 05:00:41,394 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN 2023-02-13 05:00:41,394 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN 2023-02-13 05:00:41,395 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:41,395 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=69, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,34363,1676264434089; forceNewPlan=false, retain=true 2023-02-13 05:00:41,395 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:41,395 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:41,395 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=68, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42589,1676264434015; forceNewPlan=false, retain=true 2023-02-13 05:00:41,395 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:41,395 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=66, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42589,1676264434015; forceNewPlan=false, retain=true 2023-02-13 05:00:41,396 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=67, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39885,1676264434059; forceNewPlan=false, retain=true 2023-02-13 05:00:41,396 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:41,396 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN 2023-02-13 05:00:41,396 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=65, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,34363,1676264434089; forceNewPlan=false, retain=true 2023-02-13 05:00:41,397 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:41,397 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=64, ppid=63, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39885,1676264434059; forceNewPlan=false, retain=true 2023-02-13 05:00:41,545 INFO [jenkins-hbase4:45905] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 05:00:41,545 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:41,545 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:41,545 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:41,545 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:41,546 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264441545"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264441545"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264441545"}]},"ts":"1676264441545"} 2023-02-13 05:00:41,545 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:41,546 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264441545"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264441545"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264441545"}]},"ts":"1676264441545"} 2023-02-13 05:00:41,546 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264441545"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264441545"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264441545"}]},"ts":"1676264441545"} 2023-02-13 05:00:41,546 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264441545"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264441545"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264441545"}]},"ts":"1676264441545"} 2023-02-13 05:00:41,546 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264441545"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264441545"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264441545"}]},"ts":"1676264441545"} 2023-02-13 05:00:41,548 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=70, ppid=69, state=RUNNABLE; OpenRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:41,549 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=71, ppid=66, state=RUNNABLE; OpenRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:41,550 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=72, ppid=67, state=RUNNABLE; OpenRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:41,551 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=73, ppid=68, state=RUNNABLE; OpenRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:41,552 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=74, ppid=64, state=RUNNABLE; OpenRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:41,553 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:41,553 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264441553"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264441553"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264441553"}]},"ts":"1676264441553"} 2023-02-13 05:00:41,557 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=75, ppid=65, state=RUNNABLE; OpenRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:41,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-13 05:00:41,707 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:41,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 11ed57e84fcc1f7d47e9ad2ed09021f8, NAME => 'testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 05:00:41,707 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:41,707 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:41,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 347ea8719201fbf1ee1e731b16986087, NAME => 'testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 05:00:41,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7017e1f9205c2ebde9f23df4733114d5, NAME => 'testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 05:00:41,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:41,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:41,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:41,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:41,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:41,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:41,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:41,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:41,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:41,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:41,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:41,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:41,711 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:41,711 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:41,711 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:41,713 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf 2023-02-13 05:00:41,714 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf 2023-02-13 05:00:41,714 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf 2023-02-13 05:00:41,714 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf 2023-02-13 05:00:41,714 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf 2023-02-13 05:00:41,714 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf 2023-02-13 05:00:41,714 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7017e1f9205c2ebde9f23df4733114d5 columnFamilyName cf 2023-02-13 05:00:41,714 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 347ea8719201fbf1ee1e731b16986087 columnFamilyName cf 2023-02-13 05:00:41,715 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 11ed57e84fcc1f7d47e9ad2ed09021f8 columnFamilyName cf 2023-02-13 05:00:41,725 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/519efdc2aee8432c80befe61da0da81c 2023-02-13 05:00:41,725 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d15ef8effb63426f88cf28b863b418da 2023-02-13 05:00:41,725 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/6414549fb99340299c9cb4b42e13bd55 2023-02-13 05:00:41,726 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(310): Store=11ed57e84fcc1f7d47e9ad2ed09021f8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:41,726 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(310): Store=347ea8719201fbf1ee1e731b16986087/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:41,726 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(310): Store=7017e1f9205c2ebde9f23df4733114d5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:41,727 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:41,727 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:41,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:41,729 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:41,732 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:41,733 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:41,733 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:41,733 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:41,734 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 11ed57e84fcc1f7d47e9ad2ed09021f8; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61481597, jitterRate=-0.08385281264781952}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:41,735 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 11ed57e84fcc1f7d47e9ad2ed09021f8: Running coprocessor pre-open hook at 1676264441708Writing region info on filesystem at 1676264441708Initializing all the Stores at 1676264441711 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264441711Cleaning up temporary data from old regions at 1676264441729 (+18 ms)Cleaning up detritus from prior splits at 1676264441730 (+1 ms)Running coprocessor post-open hooks at 1676264441735 (+5 ms)Region opened successfully at 1676264441735 2023-02-13 05:00:41,735 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7017e1f9205c2ebde9f23df4733114d5; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72386650, jitterRate=0.07864513993263245}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:41,735 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7017e1f9205c2ebde9f23df4733114d5: Running coprocessor pre-open hook at 1676264441708Writing region info on filesystem at 1676264441708Initializing all the Stores at 1676264441711 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264441711Cleaning up temporary data from old regions at 1676264441729 (+18 ms)Cleaning up detritus from prior splits at 1676264441730 (+1 ms)Running coprocessor post-open hooks at 1676264441735 (+5 ms)Region opened successfully at 1676264441735 2023-02-13 05:00:41,736 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8., pid=71, masterSystemTime=1676264441701 2023-02-13 05:00:41,736 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5., pid=70, masterSystemTime=1676264441701 2023-02-13 05:00:41,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:41,739 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:41,739 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:41,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:41,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ee2d32fb6918e03288e862be581d9bcb, NAME => 'testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 05:00:41,740 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=66 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:41,740 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264441739"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264441739"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264441739"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264441739"}]},"ts":"1676264441739"} 2023-02-13 05:00:41,740 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:41,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:41,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:41,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:41,741 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 347ea8719201fbf1ee1e731b16986087; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74791689, jitterRate=0.11448301374912262}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:41,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 347ea8719201fbf1ee1e731b16986087: Running coprocessor pre-open hook at 1676264441708Writing region info on filesystem at 1676264441708Initializing all the Stores at 1676264441711 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264441711Cleaning up temporary data from old regions at 1676264441734 (+23 ms)Cleaning up detritus from prior splits at 1676264441736 (+2 ms)Running coprocessor post-open hooks at 1676264441741 (+5 ms)Region opened successfully at 1676264441741 2023-02-13 05:00:41,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:41,742 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=69 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:41,741 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:41,742 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:41,742 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264441742"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264441742"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264441742"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264441742"}]},"ts":"1676264441742"} 2023-02-13 05:00:41,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d063b19c8c76cc04b197243b846f83ab, NAME => 'testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 05:00:41,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:41,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:41,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:41,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:41,744 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087., pid=72, masterSystemTime=1676264441703 2023-02-13 05:00:41,745 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:41,746 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:41,746 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf 2023-02-13 05:00:41,746 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf 2023-02-13 05:00:41,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:41,747 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:41,747 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:41,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d429d785f30801586e3db3ae3a9428b8, NAME => 'testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 05:00:41,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:41,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:41,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:41,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:41,748 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d063b19c8c76cc04b197243b846f83ab columnFamilyName cf 2023-02-13 05:00:41,748 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf 2023-02-13 05:00:41,748 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf 2023-02-13 05:00:41,747 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=71, resume processing ppid=66 2023-02-13 05:00:41,748 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=71, ppid=66, state=SUCCESS; OpenRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,42589,1676264434015 in 194 msec 2023-02-13 05:00:41,749 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee2d32fb6918e03288e862be581d9bcb columnFamilyName cf 2023-02-13 05:00:41,749 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=67 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:41,749 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264441749"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264441749"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264441749"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264441749"}]},"ts":"1676264441749"} 2023-02-13 05:00:41,750 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=70, resume processing ppid=69 2023-02-13 05:00:41,750 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=70, ppid=69, state=SUCCESS; OpenRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,34363,1676264434089 in 196 msec 2023-02-13 05:00:41,750 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=66, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN in 356 msec 2023-02-13 05:00:41,752 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=69, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN in 359 msec 2023-02-13 05:00:41,752 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:41,753 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=72, resume processing ppid=67 2023-02-13 05:00:41,753 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=72, ppid=67, state=SUCCESS; OpenRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,39885,1676264434059 in 201 msec 2023-02-13 05:00:41,754 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf 2023-02-13 05:00:41,754 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf 2023-02-13 05:00:41,755 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d429d785f30801586e3db3ae3a9428b8 columnFamilyName cf 2023-02-13 05:00:41,755 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=67, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN in 362 msec 2023-02-13 05:00:41,761 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/5ba29d67350f40888be6bfe3fec7f441 2023-02-13 05:00:41,762 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(310): Store=d063b19c8c76cc04b197243b846f83ab/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:41,763 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:41,763 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/d3866b21c6944cf0882c65fec182cb40 2023-02-13 05:00:41,763 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(310): Store=ee2d32fb6918e03288e862be581d9bcb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:41,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:41,764 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:41,765 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/811ac31223674808858dc07f9153be03 2023-02-13 05:00:41,765 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(310): Store=d429d785f30801586e3db3ae3a9428b8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:41,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:41,767 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:41,768 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:41,769 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:41,770 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d063b19c8c76cc04b197243b846f83ab; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74661494, jitterRate=0.11254295706748962}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:41,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d063b19c8c76cc04b197243b846f83ab: Running coprocessor pre-open hook at 1676264441743Writing region info on filesystem at 1676264441743Initializing all the Stores at 1676264441744 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264441744Cleaning up temporary data from old regions at 1676264441765 (+21 ms)Cleaning up detritus from prior splits at 1676264441766 (+1 ms)Running coprocessor post-open hooks at 1676264441770 (+4 ms)Region opened successfully at 1676264441770 2023-02-13 05:00:41,770 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:41,771 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab., pid=75, masterSystemTime=1676264441701 2023-02-13 05:00:41,772 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ee2d32fb6918e03288e862be581d9bcb; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72071314, jitterRate=0.0739462673664093}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:41,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ee2d32fb6918e03288e862be581d9bcb: Running coprocessor pre-open hook at 1676264441743Writing region info on filesystem at 1676264441743Initializing all the Stores at 1676264441745 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264441745Cleaning up temporary data from old regions at 1676264441767 (+22 ms)Cleaning up detritus from prior splits at 1676264441768 (+1 ms)Running coprocessor post-open hooks at 1676264441772 (+4 ms)Region opened successfully at 1676264441772 2023-02-13 05:00:41,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:41,773 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb., pid=73, masterSystemTime=1676264441701 2023-02-13 05:00:41,773 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:41,774 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d429d785f30801586e3db3ae3a9428b8; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64054233, jitterRate=-0.04551754891872406}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:41,774 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:41,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d429d785f30801586e3db3ae3a9428b8: Running coprocessor pre-open hook at 1676264441748Writing region info on filesystem at 1676264441748Initializing all the Stores at 1676264441749 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264441749Cleaning up temporary data from old regions at 1676264441769 (+20 ms)Cleaning up detritus from prior splits at 1676264441769Running coprocessor post-open hooks at 1676264441774 (+5 ms)Region opened successfully at 1676264441774 2023-02-13 05:00:41,775 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=65 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:41,775 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264441774"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264441774"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264441774"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264441774"}]},"ts":"1676264441774"} 2023-02-13 05:00:41,776 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8., pid=74, masterSystemTime=1676264441703 2023-02-13 05:00:41,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:41,776 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:41,777 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=68 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:41,777 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264441777"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264441777"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264441777"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264441777"}]},"ts":"1676264441777"} 2023-02-13 05:00:41,778 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:41,779 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:41,779 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=64 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:41,779 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264441779"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264441779"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264441779"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264441779"}]},"ts":"1676264441779"} 2023-02-13 05:00:41,780 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=75, resume processing ppid=65 2023-02-13 05:00:41,780 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=75, ppid=65, state=SUCCESS; OpenRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,34363,1676264434089 in 222 msec 2023-02-13 05:00:41,782 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=65, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN in 389 msec 2023-02-13 05:00:41,782 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=73, resume processing ppid=68 2023-02-13 05:00:41,783 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=73, ppid=68, state=SUCCESS; OpenRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,42589,1676264434015 in 228 msec 2023-02-13 05:00:41,784 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=74, resume processing ppid=64 2023-02-13 05:00:41,784 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=68, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN in 392 msec 2023-02-13 05:00:41,784 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=74, ppid=64, state=SUCCESS; OpenRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059 in 229 msec 2023-02-13 05:00:41,786 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-02-13 05:00:41,786 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN in 393 msec 2023-02-13 05:00:41,787 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264441787"}]},"ts":"1676264441787"} 2023-02-13 05:00:41,789 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=ENABLED in hbase:meta 2023-02-13 05:00:41,790 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676264436786' was successfully enabled. 2023-02-13 05:00:41,792 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676264436786 in 414 msec 2023-02-13 05:00:42,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=63 2023-02-13 05:00:42,134 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676264436786, procId: 63 completed 2023-02-13 05:00:42,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:00:42,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:00:42,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:00:42,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:00:42,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:00:42,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HRegion(8801): writing data to region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:00:42,161 DEBUG [Listener at localhost/42165] hbase.HBaseTestingUtility(2627): Found 6 regions for table testRestoreSnapshot-1676264436786 2023-02-13 05:00:42,161 DEBUG [Listener at localhost/42165] hbase.HBaseTestingUtility(2633): firstRegionName=testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:42,163 INFO [Listener at localhost/42165] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 05:00:42,405 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 05:00:42,550 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 05:00:42,675 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 05:00:42,840 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 05:00:42,977 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 05:00:43,052 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 05:00:43,053 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-02-13 05:00:44,405 DEBUG [Listener at localhost/42165] zookeeper.ReadOnlyZKClient(139): Connect 0x6eb1f1d7 to 127.0.0.1:63943 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 05:00:44,411 DEBUG [Listener at localhost/42165] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2e22f50a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 05:00:44,413 DEBUG [hconnection-0x5d7d58c4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:44,416 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44506, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:44,540 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 05:00:44,544 DEBUG [hconnection-0x5d7d58c4-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:44,549 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41008, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:44,650 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 05:00:44,654 DEBUG [hconnection-0x5d7d58c4-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:44,658 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58208, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:44,748 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 05:00:44,873 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 05:00:44,978 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 05:00:46,111 INFO [Listener at localhost/42165] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676264436786 2023-02-13 05:00:46,112 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676264436786 2023-02-13 05:00:46,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=76, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:46,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-13 05:00:46,117 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264446116"}]},"ts":"1676264446116"} 2023-02-13 05:00:46,118 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=DISABLING in hbase:meta 2023-02-13 05:00:46,120 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676264436786 to state=DISABLING 2023-02-13 05:00:46,121 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN}, {pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN}, {pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN}, {pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN}, {pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN}, {pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN}] 2023-02-13 05:00:46,123 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=78, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN 2023-02-13 05:00:46,123 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=79, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN 2023-02-13 05:00:46,123 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=77, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN 2023-02-13 05:00:46,123 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=80, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN 2023-02-13 05:00:46,123 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=82, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN 2023-02-13 05:00:46,124 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:46,124 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:46,124 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:46,124 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264446124"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264446124"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264446124"}]},"ts":"1676264446124"} 2023-02-13 05:00:46,124 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264446124"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264446124"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264446124"}]},"ts":"1676264446124"} 2023-02-13 05:00:46,124 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264446124"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264446124"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264446124"}]},"ts":"1676264446124"} 2023-02-13 05:00:46,124 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:46,125 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:46,125 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264446124"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264446124"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264446124"}]},"ts":"1676264446124"} 2023-02-13 05:00:46,125 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264446124"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264446124"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264446124"}]},"ts":"1676264446124"} 2023-02-13 05:00:46,126 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=83, ppid=77, state=RUNNABLE; CloseRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:46,126 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=84, ppid=78, state=RUNNABLE; CloseRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:46,127 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=85, ppid=79, state=RUNNABLE; CloseRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:46,128 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=86, ppid=80, state=RUNNABLE; CloseRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:46,129 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=87, ppid=82, state=RUNNABLE; CloseRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:46,130 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=81, ppid=76, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN 2023-02-13 05:00:46,131 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:46,131 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264446131"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264446131"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264446131"}]},"ts":"1676264446131"} 2023-02-13 05:00:46,133 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=88, ppid=81, state=RUNNABLE; CloseRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:46,278 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:46,278 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:46,278 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 347ea8719201fbf1ee1e731b16986087, disabling compactions & flushes 2023-02-13 05:00:46,279 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d063b19c8c76cc04b197243b846f83ab, disabling compactions & flushes 2023-02-13 05:00:46,279 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:46,279 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:46,279 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:46,279 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:46,279 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. after waiting 0 ms 2023-02-13 05:00:46,280 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. after waiting 0 ms 2023-02-13 05:00:46,280 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:46,280 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:46,280 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing d063b19c8c76cc04b197243b846f83ab 1/1 column families, dataSize=1.57 KB heapSize=3.63 KB 2023-02-13 05:00:46,280 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 347ea8719201fbf1ee1e731b16986087 1/1 column families, dataSize=1.96 KB heapSize=4.47 KB 2023-02-13 05:00:46,280 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:46,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 11ed57e84fcc1f7d47e9ad2ed09021f8, disabling compactions & flushes 2023-02-13 05:00:46,281 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:46,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:46,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. after waiting 0 ms 2023-02-13 05:00:46,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:46,281 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 11ed57e84fcc1f7d47e9ad2ed09021f8 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-13 05:00:46,317 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 2023-02-13 05:00:46,317 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c 2023-02-13 05:00:46,317 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 2023-02-13 05:00:46,328 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/.tmp/cf/075fd0e105614d9aaf0995ac7325a2fb 2023-02-13 05:00:46,329 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/.tmp/cf/b0b32b9f22144056b97173b2a69002a5 2023-02-13 05:00:46,330 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/.tmp/cf/3ae4957471034eeb9ed26ec1a046807b 2023-02-13 05:00:46,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/.tmp/cf/b0b32b9f22144056b97173b2a69002a5 as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/b0b32b9f22144056b97173b2a69002a5 2023-02-13 05:00:46,333 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/.tmp/cf/075fd0e105614d9aaf0995ac7325a2fb as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/075fd0e105614d9aaf0995ac7325a2fb 2023-02-13 05:00:46,334 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/.tmp/cf/3ae4957471034eeb9ed26ec1a046807b as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/3ae4957471034eeb9ed26ec1a046807b 2023-02-13 05:00:46,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/075fd0e105614d9aaf0995ac7325a2fb, entries=24, sequenceid=15, filesize=9.0 K 2023-02-13 05:00:46,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/3ae4957471034eeb9ed26ec1a046807b, entries=36, sequenceid=15, filesize=11.0 K 2023-02-13 05:00:46,342 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/b0b32b9f22144056b97173b2a69002a5, entries=30, sequenceid=15, filesize=10.0 K 2023-02-13 05:00:46,342 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 11ed57e84fcc1f7d47e9ad2ed09021f8 in 61ms, sequenceid=15, compaction requested=false 2023-02-13 05:00:46,343 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 347ea8719201fbf1ee1e731b16986087 in 63ms, sequenceid=15, compaction requested=false 2023-02-13 05:00:46,347 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.57 KB/1606, heapSize ~3.61 KB/3696, currentSize=0 B/0 for d063b19c8c76cc04b197243b846f83ab in 67ms, sequenceid=15, compaction requested=false 2023-02-13 05:00:46,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 05:00:46,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 05:00:46,360 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 05:00:46,361 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:46,361 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:46,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d063b19c8c76cc04b197243b846f83ab: Waiting for close lock at 1676264446278Running coprocessor pre-close hooks at 1676264446278Disabling compacts and flushes for region at 1676264446278Disabling writes for close at 1676264446280 (+2 ms)Obtaining lock to block concurrent updates at 1676264446280Preparing flush snapshotting stores in d063b19c8c76cc04b197243b846f83ab at 1676264446280Finished memstore snapshotting testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab., syncing WAL and waiting on mvcc, flushsize=dataSize=1606, getHeapSize=3696, getOffHeapSize=0, getCellsCount=24 at 1676264446280Flushing stores of testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. at 1676264446282 (+2 ms)Flushing d063b19c8c76cc04b197243b846f83ab/cf: creating writer at 1676264446282Flushing mob file d063b19c8c76cc04b197243b846f83ab/cf: appending metadata at 1676264446291 (+9 ms)Flushing mob file d063b19c8c76cc04b197243b846f83ab/cf: closing flushed file at 1676264446291Flushing d063b19c8c76cc04b197243b846f83ab/cf: appending metadata at 1676264446318 (+27 ms)Flushing d063b19c8c76cc04b197243b846f83ab/cf: closing flushed file at 1676264446318Flushing d063b19c8c76cc04b197243b846f83ab/cf: reopening flushed file at 1676264446334 (+16 ms)Finished flush of dataSize ~1.57 KB/1606, heapSize ~3.61 KB/3696, currentSize=0 B/0 for d063b19c8c76cc04b197243b846f83ab in 67ms, sequenceid=15, compaction requested=false at 1676264446347 (+13 ms)Writing region close event to WAL at 1676264446357 (+10 ms)Running coprocessor post-close hooks at 1676264446361 (+4 ms)Closed at 1676264446361 2023-02-13 05:00:46,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 347ea8719201fbf1ee1e731b16986087: Waiting for close lock at 1676264446278Running coprocessor pre-close hooks at 1676264446278Disabling compacts and flushes for region at 1676264446278Disabling writes for close at 1676264446280 (+2 ms)Obtaining lock to block concurrent updates at 1676264446280Preparing flush snapshotting stores in 347ea8719201fbf1ee1e731b16986087 at 1676264446280Finished memstore snapshotting testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087., syncing WAL and waiting on mvcc, flushsize=dataSize=2008, getHeapSize=4560, getOffHeapSize=0, getCellsCount=30 at 1676264446280Flushing stores of testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. at 1676264446282 (+2 ms)Flushing 347ea8719201fbf1ee1e731b16986087/cf: creating writer at 1676264446283 (+1 ms)Flushing mob file 347ea8719201fbf1ee1e731b16986087/cf: appending metadata at 1676264446291 (+8 ms)Flushing mob file 347ea8719201fbf1ee1e731b16986087/cf: closing flushed file at 1676264446291Flushing 347ea8719201fbf1ee1e731b16986087/cf: appending metadata at 1676264446318 (+27 ms)Flushing 347ea8719201fbf1ee1e731b16986087/cf: closing flushed file at 1676264446318Flushing 347ea8719201fbf1ee1e731b16986087/cf: reopening flushed file at 1676264446334 (+16 ms)Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for 347ea8719201fbf1ee1e731b16986087 in 63ms, sequenceid=15, compaction requested=false at 1676264446344 (+10 ms)Writing region close event to WAL at 1676264446355 (+11 ms)Running coprocessor post-close hooks at 1676264446361 (+6 ms)Closed at 1676264446361 2023-02-13 05:00:46,361 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:46,361 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 11ed57e84fcc1f7d47e9ad2ed09021f8: Waiting for close lock at 1676264446280Running coprocessor pre-close hooks at 1676264446280Disabling compacts and flushes for region at 1676264446280Disabling writes for close at 1676264446281 (+1 ms)Obtaining lock to block concurrent updates at 1676264446281Preparing flush snapshotting stores in 11ed57e84fcc1f7d47e9ad2ed09021f8 at 1676264446281Finished memstore snapshotting testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8., syncing WAL and waiting on mvcc, flushsize=dataSize=2410, getHeapSize=5424, getOffHeapSize=0, getCellsCount=36 at 1676264446282 (+1 ms)Flushing stores of testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. at 1676264446284 (+2 ms)Flushing 11ed57e84fcc1f7d47e9ad2ed09021f8/cf: creating writer at 1676264446284Flushing mob file 11ed57e84fcc1f7d47e9ad2ed09021f8/cf: appending metadata at 1676264446291 (+7 ms)Flushing mob file 11ed57e84fcc1f7d47e9ad2ed09021f8/cf: closing flushed file at 1676264446291Flushing 11ed57e84fcc1f7d47e9ad2ed09021f8/cf: appending metadata at 1676264446318 (+27 ms)Flushing 11ed57e84fcc1f7d47e9ad2ed09021f8/cf: closing flushed file at 1676264446318Flushing 11ed57e84fcc1f7d47e9ad2ed09021f8/cf: reopening flushed file at 1676264446338 (+20 ms)Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 11ed57e84fcc1f7d47e9ad2ed09021f8 in 61ms, sequenceid=15, compaction requested=false at 1676264446343 (+5 ms)Writing region close event to WAL at 1676264446353 (+10 ms)Running coprocessor post-close hooks at 1676264446361 (+8 ms)Closed at 1676264446361 2023-02-13 05:00:46,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:46,363 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:46,363 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=78 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=CLOSED 2023-02-13 05:00:46,364 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264446363"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264446363"}]},"ts":"1676264446363"} 2023-02-13 05:00:46,364 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7017e1f9205c2ebde9f23df4733114d5, disabling compactions & flushes 2023-02-13 05:00:46,364 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:46,365 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:46,365 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. after waiting 0 ms 2023-02-13 05:00:46,365 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:46,365 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:46,365 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:46,365 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d429d785f30801586e3db3ae3a9428b8, disabling compactions & flushes 2023-02-13 05:00:46,365 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 7017e1f9205c2ebde9f23df4733114d5 1/1 column families, dataSize=22.63 KB heapSize=48.91 KB 2023-02-13 05:00:46,365 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:46,365 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:46,365 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. after waiting 0 ms 2023-02-13 05:00:46,365 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:46,365 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing d429d785f30801586e3db3ae3a9428b8 1/1 column families, dataSize=2.48 KB heapSize=5.59 KB 2023-02-13 05:00:46,367 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=80 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=CLOSED 2023-02-13 05:00:46,367 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264446367"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264446367"}]},"ts":"1676264446367"} 2023-02-13 05:00:46,368 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:46,368 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:46,368 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=79 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=CLOSED 2023-02-13 05:00:46,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-13 05:00:46,368 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264446368"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264446368"}]},"ts":"1676264446368"} 2023-02-13 05:00:46,372 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=84, resume processing ppid=78 2023-02-13 05:00:46,372 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=84, ppid=78, state=SUCCESS; CloseRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,34363,1676264434089 in 242 msec 2023-02-13 05:00:46,373 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=86, resume processing ppid=80 2023-02-13 05:00:46,373 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=86, ppid=80, state=SUCCESS; CloseRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,39885,1676264434059 in 242 msec 2023-02-13 05:00:46,374 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=78, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN in 251 msec 2023-02-13 05:00:46,375 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=85, resume processing ppid=79 2023-02-13 05:00:46,375 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=85, ppid=79, state=SUCCESS; CloseRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,42589,1676264434015 in 245 msec 2023-02-13 05:00:46,376 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=80, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN in 252 msec 2023-02-13 05:00:46,377 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=79, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN in 254 msec 2023-02-13 05:00:46,384 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ee2d32fb6918e03288e862be581d9bcb, disabling compactions & flushes 2023-02-13 05:00:46,384 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:46,384 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:46,384 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. after waiting 0 ms 2023-02-13 05:00:46,384 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:46,384 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing ee2d32fb6918e03288e862be581d9bcb 1/1 column families, dataSize=1.70 KB heapSize=3.91 KB 2023-02-13 05:00:46,403 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 2023-02-13 05:00:46,418 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b 2023-02-13 05:00:46,428 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/.tmp/cf/b95f98eee2d241bea094905381835a03 2023-02-13 05:00:46,432 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 2023-02-13 05:00:46,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/.tmp/cf/b95f98eee2d241bea094905381835a03 as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/b95f98eee2d241bea094905381835a03 2023-02-13 05:00:46,436 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=22.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/.tmp/cf/d318209d74c44b3fb2d00363c5890672 2023-02-13 05:00:46,438 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/b95f98eee2d241bea094905381835a03, entries=38, sequenceid=15, filesize=11.4 K 2023-02-13 05:00:46,439 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.48 KB/2544, heapSize ~5.58 KB/5712, currentSize=0 B/0 for d429d785f30801586e3db3ae3a9428b8 in 73ms, sequenceid=15, compaction requested=false 2023-02-13 05:00:46,448 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/.tmp/cf/d318209d74c44b3fb2d00363c5890672 as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d318209d74c44b3fb2d00363c5890672 2023-02-13 05:00:46,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 05:00:46,450 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:46,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d429d785f30801586e3db3ae3a9428b8: Waiting for close lock at 1676264446365Running coprocessor pre-close hooks at 1676264446365Disabling compacts and flushes for region at 1676264446365Disabling writes for close at 1676264446365Obtaining lock to block concurrent updates at 1676264446365Preparing flush snapshotting stores in d429d785f30801586e3db3ae3a9428b8 at 1676264446365Finished memstore snapshotting testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8., syncing WAL and waiting on mvcc, flushsize=dataSize=2544, getHeapSize=5712, getOffHeapSize=0, getCellsCount=38 at 1676264446365Flushing stores of testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. at 1676264446366 (+1 ms)Flushing d429d785f30801586e3db3ae3a9428b8/cf: creating writer at 1676264446367 (+1 ms)Flushing mob file d429d785f30801586e3db3ae3a9428b8/cf: appending metadata at 1676264446385 (+18 ms)Flushing mob file d429d785f30801586e3db3ae3a9428b8/cf: closing flushed file at 1676264446385Flushing d429d785f30801586e3db3ae3a9428b8/cf: appending metadata at 1676264446404 (+19 ms)Flushing d429d785f30801586e3db3ae3a9428b8/cf: closing flushed file at 1676264446404Flushing d429d785f30801586e3db3ae3a9428b8/cf: reopening flushed file at 1676264446434 (+30 ms)Finished flush of dataSize ~2.48 KB/2544, heapSize ~5.58 KB/5712, currentSize=0 B/0 for d429d785f30801586e3db3ae3a9428b8 in 73ms, sequenceid=15, compaction requested=false at 1676264446439 (+5 ms)Writing region close event to WAL at 1676264446443 (+4 ms)Running coprocessor post-close hooks at 1676264446450 (+7 ms)Closed at 1676264446450 2023-02-13 05:00:46,452 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:46,452 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d318209d74c44b3fb2d00363c5890672, entries=346, sequenceid=15, filesize=63.8 K 2023-02-13 05:00:46,453 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=77 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=CLOSED 2023-02-13 05:00:46,453 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.63 KB/23178, heapSize ~48.89 KB/50064, currentSize=0 B/0 for 7017e1f9205c2ebde9f23df4733114d5 in 88ms, sequenceid=15, compaction requested=false 2023-02-13 05:00:46,453 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264446453"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264446453"}]},"ts":"1676264446453"} 2023-02-13 05:00:46,459 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 05:00:46,459 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=83, resume processing ppid=77 2023-02-13 05:00:46,460 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=83, ppid=77, state=SUCCESS; CloseRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059 in 331 msec 2023-02-13 05:00:46,460 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:46,460 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7017e1f9205c2ebde9f23df4733114d5: Waiting for close lock at 1676264446363Running coprocessor pre-close hooks at 1676264446363Disabling compacts and flushes for region at 1676264446363Disabling writes for close at 1676264446365 (+2 ms)Obtaining lock to block concurrent updates at 1676264446365Preparing flush snapshotting stores in 7017e1f9205c2ebde9f23df4733114d5 at 1676264446365Finished memstore snapshotting testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5., syncing WAL and waiting on mvcc, flushsize=dataSize=23178, getHeapSize=50064, getOffHeapSize=0, getCellsCount=346 at 1676264446365Flushing stores of testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. at 1676264446366 (+1 ms)Flushing 7017e1f9205c2ebde9f23df4733114d5/cf: creating writer at 1676264446367 (+1 ms)Flushing mob file 7017e1f9205c2ebde9f23df4733114d5/cf: appending metadata at 1676264446388 (+21 ms)Flushing mob file 7017e1f9205c2ebde9f23df4733114d5/cf: closing flushed file at 1676264446389 (+1 ms)Flushing 7017e1f9205c2ebde9f23df4733114d5/cf: appending metadata at 1676264446418 (+29 ms)Flushing 7017e1f9205c2ebde9f23df4733114d5/cf: closing flushed file at 1676264446419 (+1 ms)Flushing 7017e1f9205c2ebde9f23df4733114d5/cf: reopening flushed file at 1676264446449 (+30 ms)Finished flush of dataSize ~22.63 KB/23178, heapSize ~48.89 KB/50064, currentSize=0 B/0 for 7017e1f9205c2ebde9f23df4733114d5 in 88ms, sequenceid=15, compaction requested=false at 1676264446453 (+4 ms)Writing region close event to WAL at 1676264446456 (+3 ms)Running coprocessor post-close hooks at 1676264446460 (+4 ms)Closed at 1676264446460 2023-02-13 05:00:46,463 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:46,463 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=77, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN in 338 msec 2023-02-13 05:00:46,463 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=82 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=CLOSED 2023-02-13 05:00:46,464 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264446463"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264446463"}]},"ts":"1676264446463"} 2023-02-13 05:00:46,467 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=87, resume processing ppid=82 2023-02-13 05:00:46,467 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=87, ppid=82, state=SUCCESS; CloseRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,34363,1676264434089 in 336 msec 2023-02-13 05:00:46,468 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=82, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN in 346 msec 2023-02-13 05:00:46,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=1.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/.tmp/cf/3f25ec8511e84234a9c7a722ca070d0b 2023-02-13 05:00:46,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/.tmp/cf/3f25ec8511e84234a9c7a722ca070d0b as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/3f25ec8511e84234a9c7a722ca070d0b 2023-02-13 05:00:46,858 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/3f25ec8511e84234a9c7a722ca070d0b, entries=26, sequenceid=15, filesize=9.3 K 2023-02-13 05:00:46,859 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for ee2d32fb6918e03288e862be581d9bcb in 475ms, sequenceid=15, compaction requested=false 2023-02-13 05:00:46,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 05:00:46,866 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:46,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ee2d32fb6918e03288e862be581d9bcb: Waiting for close lock at 1676264446368Running coprocessor pre-close hooks at 1676264446368Disabling compacts and flushes for region at 1676264446368Disabling writes for close at 1676264446384 (+16 ms)Obtaining lock to block concurrent updates at 1676264446384Preparing flush snapshotting stores in ee2d32fb6918e03288e862be581d9bcb at 1676264446384Finished memstore snapshotting testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb., syncing WAL and waiting on mvcc, flushsize=dataSize=1740, getHeapSize=3984, getOffHeapSize=0, getCellsCount=26 at 1676264446384Flushing stores of testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. at 1676264446385 (+1 ms)Flushing ee2d32fb6918e03288e862be581d9bcb/cf: creating writer at 1676264446386 (+1 ms)Flushing mob file ee2d32fb6918e03288e862be581d9bcb/cf: appending metadata at 1676264446394 (+8 ms)Flushing mob file ee2d32fb6918e03288e862be581d9bcb/cf: closing flushed file at 1676264446394Flushing ee2d32fb6918e03288e862be581d9bcb/cf: appending metadata at 1676264446433 (+39 ms)Flushing ee2d32fb6918e03288e862be581d9bcb/cf: closing flushed file at 1676264446433Flushing ee2d32fb6918e03288e862be581d9bcb/cf: reopening flushed file at 1676264446854 (+421 ms)Finished flush of dataSize ~1.70 KB/1740, heapSize ~3.89 KB/3984, currentSize=0 B/0 for ee2d32fb6918e03288e862be581d9bcb in 475ms, sequenceid=15, compaction requested=false at 1676264446859 (+5 ms)Writing region close event to WAL at 1676264446862 (+3 ms)Running coprocessor post-close hooks at 1676264446866 (+4 ms)Closed at 1676264446866 2023-02-13 05:00:46,868 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:46,869 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=81 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=CLOSED 2023-02-13 05:00:46,869 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264446869"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264446869"}]},"ts":"1676264446869"} 2023-02-13 05:00:46,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-13 05:00:46,872 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=88, resume processing ppid=81 2023-02-13 05:00:46,872 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=88, ppid=81, state=SUCCESS; CloseRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,42589,1676264434015 in 737 msec 2023-02-13 05:00:46,874 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=81, resume processing ppid=76 2023-02-13 05:00:46,874 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=81, ppid=76, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN in 751 msec 2023-02-13 05:00:46,875 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264446875"}]},"ts":"1676264446875"} 2023-02-13 05:00:46,876 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=DISABLED in hbase:meta 2023-02-13 05:00:46,880 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676264436786 to state=DISABLED 2023-02-13 05:00:46,882 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=76, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676264436786 in 768 msec 2023-02-13 05:00:47,211 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 05:00:47,283 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6eb1f1d7 to 127.0.0.1:63943 2023-02-13 05:00:47,284 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:00:47,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=76 2023-02-13 05:00:47,621 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676264436786, procId: 76 completed 2023-02-13 05:00:47,622 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb1-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } 2023-02-13 05:00:47,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676264447622 (current time:1676264447622). 2023-02-13 05:00:47,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 05:00:47,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] zookeeper.ReadOnlyZKClient(139): Connect 0x3f73be94 to 127.0.0.1:63943 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 05:00:47,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@184df5cf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 05:00:47,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:47,631 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44518, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:47,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3f73be94 to 127.0.0.1:63943 2023-02-13 05:00:47,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:00:47,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 05:00:47,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb1-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } 2023-02-13 05:00:47,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE 2023-02-13 05:00:47,635 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=89, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE 2023-02-13 05:00:47,636 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=89, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE; timeout=600000, timestamp=1676265047636 2023-02-13 05:00:47,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb1-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } 2023-02-13 05:00:47,636 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb1-1676264436786 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676264436786 2023-02-13 05:00:47,636 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb1-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 05:00:47,636 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 05:00:47,649 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676264436786 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } 2023-02-13 05:00:47,657 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:47,657 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:47,657 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:47,657 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:47,657 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:47,657 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:47,657 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:47,657 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:47,658 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:47,658 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:47,658 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:47,658 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:47,658 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:47,658 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:47,660 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/519efdc2aee8432c80befe61da0da81c, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/b0b32b9f22144056b97173b2a69002a5] hfiles 2023-02-13 05:00:47,660 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/519efdc2aee8432c80befe61da0da81c 2023-02-13 05:00:47,660 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/b0b32b9f22144056b97173b2a69002a5 2023-02-13 05:00:47,661 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/075fd0e105614d9aaf0995ac7325a2fb, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/5ba29d67350f40888be6bfe3fec7f441] hfiles 2023-02-13 05:00:47,661 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/075fd0e105614d9aaf0995ac7325a2fb 2023-02-13 05:00:47,661 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/5ba29d67350f40888be6bfe3fec7f441 2023-02-13 05:00:47,665 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d15ef8effb63426f88cf28b863b418da, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d318209d74c44b3fb2d00363c5890672] hfiles 2023-02-13 05:00:47,665 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d15ef8effb63426f88cf28b863b418da 2023-02-13 05:00:47,665 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d318209d74c44b3fb2d00363c5890672 2023-02-13 05:00:47,665 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/3f25ec8511e84234a9c7a722ca070d0b, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/d3866b21c6944cf0882c65fec182cb40] hfiles 2023-02-13 05:00:47,665 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/3f25ec8511e84234a9c7a722ca070d0b 2023-02-13 05:00:47,665 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/d3866b21c6944cf0882c65fec182cb40 2023-02-13 05:00:47,666 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/811ac31223674808858dc07f9153be03, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/b95f98eee2d241bea094905381835a03] hfiles 2023-02-13 05:00:47,666 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/811ac31223674808858dc07f9153be03 2023-02-13 05:00:47,666 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/b95f98eee2d241bea094905381835a03 2023-02-13 05:00:47,668 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/3ae4957471034eeb9ed26ec1a046807b, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/6414549fb99340299c9cb4b42e13bd55] hfiles 2023-02-13 05:00:47,668 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/3ae4957471034eeb9ed26ec1a046807b 2023-02-13 05:00:47,668 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/6414549fb99340299c9cb4b42e13bd55 2023-02-13 05:00:47,669 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2] hfiles 2023-02-13 05:00:47,669 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 2023-02-13 05:00:47,669 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 2023-02-13 05:00:47,669 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 2023-02-13 05:00:47,669 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 2023-02-13 05:00:47,669 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c 2023-02-13 05:00:47,669 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 2023-02-13 05:00:47,669 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 2023-02-13 05:00:47,669 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 2023-02-13 05:00:47,669 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 2023-02-13 05:00:47,669 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b 2023-02-13 05:00:47,669 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 2023-02-13 05:00:47,669 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 2023-02-13 05:00:47,707 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb1-1676264436786 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } as finished. 2023-02-13 05:00:47,707 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb1-1676264436786 2023-02-13 05:00:47,708 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/snaptb1-1676264436786 2023-02-13 05:00:47,733 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/snaptb1-1676264436786 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/snaptb1-1676264436786 2023-02-13 05:00:47,752 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb1-1676264436786 of table testRestoreSnapshot-1676264436786 completed 2023-02-13 05:00:47,752 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/snaptb1-1676264436786 2023-02-13 05:00:47,752 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/snaptb1-1676264436786 2023-02-13 05:00:47,752 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb1-1676264436786 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676264436786 at 1676264447636Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb1-1676264436786 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } at 1676264447649 (+13 ms)Consolidate snapshot: snaptb1-1676264436786 at 1676264447707 (+58 ms)Loading Region manifests for snaptb1-1676264436786 at 1676264447707Writing data manifest for snaptb1-1676264436786 at 1676264447711 (+4 ms)Verifying snapshot: snaptb1-1676264436786 at 1676264447720 (+9 ms)Snapshot snaptb1-1676264436786 of table testRestoreSnapshot-1676264436786 completed at 1676264447752 (+32 ms) 2023-02-13 05:00:47,754 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=89, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE 2023-02-13 05:00:47,755 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=89, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE in 120 msec 2023-02-13 05:00:47,887 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 05:00:47,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb1-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } is done 2023-02-13 05:00:47,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb1-1676264436786 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 05:00:47,903 INFO [Listener at localhost/42165] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb0-1676264436786-1676264447903 2023-02-13 05:00:47,904 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb0-1676264436786-1676264447903 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } 2023-02-13 05:00:47,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676264447904 (current time:1676264447904). 2023-02-13 05:00:47,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 05:00:47,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] zookeeper.ReadOnlyZKClient(139): Connect 0x08c4cc34 to 127.0.0.1:63943 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 05:00:47,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3910fcd1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 05:00:47,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:47,914 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44534, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:47,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08c4cc34 to 127.0.0.1:63943 2023-02-13 05:00:47,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:00:47,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 05:00:47,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb0-1676264436786-1676264447903 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } 2023-02-13 05:00:47,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE 2023-02-13 05:00:47,918 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=90, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE 2023-02-13 05:00:47,918 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=90, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE; timeout=600000, timestamp=1676265047918 2023-02-13 05:00:47,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb0-1676264436786-1676264447903 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } 2023-02-13 05:00:47,918 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb0-1676264436786-1676264447903 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676264436786 2023-02-13 05:00:47,919 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb0-1676264436786-1676264447903 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 05:00:47,919 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 05:00:47,928 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676264436786-1676264447903 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } 2023-02-13 05:00:47,929 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:47,930 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:47,930 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:47,930 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:47,930 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:47,930 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:47,930 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:47,930 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:47,930 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:47,931 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:47,931 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:47,931 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:47,931 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:47,931 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:47,933 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/3f25ec8511e84234a9c7a722ca070d0b, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/d3866b21c6944cf0882c65fec182cb40] hfiles 2023-02-13 05:00:47,933 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/3f25ec8511e84234a9c7a722ca070d0b 2023-02-13 05:00:47,933 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/d3866b21c6944cf0882c65fec182cb40 2023-02-13 05:00:47,933 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/519efdc2aee8432c80befe61da0da81c, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/b0b32b9f22144056b97173b2a69002a5] hfiles 2023-02-13 05:00:47,933 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/519efdc2aee8432c80befe61da0da81c 2023-02-13 05:00:47,934 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/b0b32b9f22144056b97173b2a69002a5 2023-02-13 05:00:47,934 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/3ae4957471034eeb9ed26ec1a046807b, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/6414549fb99340299c9cb4b42e13bd55] hfiles 2023-02-13 05:00:47,934 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/3ae4957471034eeb9ed26ec1a046807b 2023-02-13 05:00:47,934 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/6414549fb99340299c9cb4b42e13bd55 2023-02-13 05:00:47,934 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d15ef8effb63426f88cf28b863b418da, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d318209d74c44b3fb2d00363c5890672] hfiles 2023-02-13 05:00:47,935 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d15ef8effb63426f88cf28b863b418da 2023-02-13 05:00:47,935 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d318209d74c44b3fb2d00363c5890672 2023-02-13 05:00:47,935 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/811ac31223674808858dc07f9153be03, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/b95f98eee2d241bea094905381835a03] hfiles 2023-02-13 05:00:47,935 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/811ac31223674808858dc07f9153be03 2023-02-13 05:00:47,935 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/b95f98eee2d241bea094905381835a03 2023-02-13 05:00:47,936 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/075fd0e105614d9aaf0995ac7325a2fb, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/5ba29d67350f40888be6bfe3fec7f441] hfiles 2023-02-13 05:00:47,937 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/075fd0e105614d9aaf0995ac7325a2fb 2023-02-13 05:00:47,937 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (2/2): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/5ba29d67350f40888be6bfe3fec7f441 2023-02-13 05:00:47,946 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2] hfiles 2023-02-13 05:00:47,947 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 2023-02-13 05:00:47,947 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 2023-02-13 05:00:47,947 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 2023-02-13 05:00:47,947 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 2023-02-13 05:00:47,947 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c 2023-02-13 05:00:47,947 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 2023-02-13 05:00:47,947 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (7/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 2023-02-13 05:00:47,947 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (8/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 2023-02-13 05:00:47,947 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (9/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 2023-02-13 05:00:47,947 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (10/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b 2023-02-13 05:00:47,947 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (11/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 2023-02-13 05:00:47,947 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (12/12): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 2023-02-13 05:00:47,973 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb0-1676264436786-1676264447903 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } as finished. 2023-02-13 05:00:47,973 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb0-1676264436786-1676264447903 2023-02-13 05:00:47,974 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676264436786-1676264447903 2023-02-13 05:00:48,003 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676264436786-1676264447903 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/hbase-failsafe-snaptb0-1676264436786-1676264447903 2023-02-13 05:00:48,026 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb0-1676264436786-1676264447903 of table testRestoreSnapshot-1676264436786 completed 2023-02-13 05:00:48,026 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676264436786-1676264447903 2023-02-13 05:00:48,026 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/hbase-failsafe-snaptb0-1676264436786-1676264447903 2023-02-13 05:00:48,026 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb0-1676264436786-1676264447903 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676264436786 at 1676264447919Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb0-1676264436786-1676264447903 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } at 1676264447928 (+9 ms)Consolidate snapshot: hbase-failsafe-snaptb0-1676264436786-1676264447903 at 1676264447973 (+45 ms)Loading Region manifests for hbase-failsafe-snaptb0-1676264436786-1676264447903 at 1676264447973Writing data manifest for hbase-failsafe-snaptb0-1676264436786-1676264447903 at 1676264447978 (+5 ms)Verifying snapshot: hbase-failsafe-snaptb0-1676264436786-1676264447903 at 1676264447991 (+13 ms)Snapshot hbase-failsafe-snaptb0-1676264436786-1676264447903 of table testRestoreSnapshot-1676264436786 completed at 1676264448026 (+35 ms) 2023-02-13 05:00:48,028 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=90, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE 2023-02-13 05:00:48,029 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=90, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE in 112 msec 2023-02-13 05:00:48,169 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 05:00:48,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb0-1676264436786-1676264447903 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } is done 2023-02-13 05:00:48,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb0-1676264436786-1676264447903 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 05:00:48,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=91, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676264436786 snapshot=name: "snaptb0-1676264436786" table: "testRestoreSnapshot-1676264436786" creation_time: 1676264441109 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 05:00:48,196 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(880): Restore snapshot=snaptb0-1676264436786 as table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:48,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-13 05:00:48,211 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/.tabledesc/.tableinfo.0000000002 2023-02-13 05:00:48,212 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/.tabledesc/.tableinfo.0000000001 2023-02-13 05:00:48,212 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/.tabledesc/.tableinfo.0000000002 2023-02-13 05:00:48,213 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb0-1676264436786 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } 2023-02-13 05:00:48,216 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb0-1676264436786" table: "testRestoreSnapshot-1676264436786" creation_time: 1676264441109 type: DISABLED version: 2 ttl: 0 2023-02-13 05:00:48,217 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786 2023-02-13 05:00:48,226 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:48,233 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c 2023-02-13 05:00:48,235 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 2023-02-13 05:00:48,236 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 2023-02-13 05:00:48,238 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 2023-02-13 05:00:48,239 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 2023-02-13 05:00:48,240 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b 2023-02-13 05:00:48,240 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:48,240 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:48,240 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:48,240 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:48,240 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:48,240 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:48,248 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/075fd0e105614d9aaf0995ac7325a2fb to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/075fd0e105614d9aaf0995ac7325a2fb 2023-02-13 05:00:48,248 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/b95f98eee2d241bea094905381835a03 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/b95f98eee2d241bea094905381835a03 2023-02-13 05:00:48,248 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/3f25ec8511e84234a9c7a722ca070d0b to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/3f25ec8511e84234a9c7a722ca070d0b 2023-02-13 05:00:48,249 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/b0b32b9f22144056b97173b2a69002a5 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/b0b32b9f22144056b97173b2a69002a5 2023-02-13 05:00:48,249 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d318209d74c44b3fb2d00363c5890672 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d318209d74c44b3fb2d00363c5890672 2023-02-13 05:00:48,249 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/3ae4957471034eeb9ed26ec1a046807b to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/3ae4957471034eeb9ed26ec1a046807b 2023-02-13 05:00:48,250 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb0-1676264436786" table: "testRestoreSnapshot-1676264436786" creation_time: 1676264441109 type: DISABLED version: 2 ttl: 0 2023-02-13 05:00:48,252 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264448251"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:48,252 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264448251"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:48,252 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264448251"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:48,252 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264448251"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:48,252 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264448251"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:48,252 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264448251"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:48,255 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 05:00:48,255 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 11ed57e84fcc1f7d47e9ad2ed09021f8, NAME => 'testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 347ea8719201fbf1ee1e731b16986087, NAME => 'testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 7017e1f9205c2ebde9f23df4733114d5, NAME => 'testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => d063b19c8c76cc04b197243b846f83ab, NAME => 'testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => d429d785f30801586e3db3ae3a9428b8, NAME => 'testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => ee2d32fb6918e03288e862be581d9bcb, NAME => 'testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.', STARTKEY => '4', ENDKEY => '5'}] 2023-02-13 05:00:48,255 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264448252"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264448252"}]},"ts":"1676264448252"} 2023-02-13 05:00:48,255 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264448252"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264448252"}]},"ts":"1676264448252"} 2023-02-13 05:00:48,256 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264448252"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264448252"}]},"ts":"1676264448252"} 2023-02-13 05:00:48,256 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264448252"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264448252"}]},"ts":"1676264448252"} 2023-02-13 05:00:48,256 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264448252"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264448252"}]},"ts":"1676264448252"} 2023-02-13 05:00:48,256 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264448252"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264448252"}]},"ts":"1676264448252"} 2023-02-13 05:00:48,258 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 05:00:48,258 INFO [PEWorker-3] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-13 05:00:48,258 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 11ed57e84fcc1f7d47e9ad2ed09021f8, NAME => 'testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 347ea8719201fbf1ee1e731b16986087, NAME => 'testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 7017e1f9205c2ebde9f23df4733114d5, NAME => 'testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => d063b19c8c76cc04b197243b846f83ab, NAME => 'testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => d429d785f30801586e3db3ae3a9428b8, NAME => 'testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => ee2d32fb6918e03288e862be581d9bcb, NAME => 'testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.', STARTKEY => '4', ENDKEY => '5'}] 2023-02-13 05:00:48,259 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb0-1676264436786 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676264436786 completed! 2023-02-13 05:00:48,261 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=91, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676264436786 snapshot=name: "snaptb0-1676264436786" table: "testRestoreSnapshot-1676264436786" creation_time: 1676264441109 type: DISABLED version: 2 ttl: 0 ) in 72 msec 2023-02-13 05:00:48,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=91 2023-02-13 05:00:48,452 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676264436786, procId: 91 completed 2023-02-13 05:00:48,452 INFO [Listener at localhost/42165] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb0-1676264436786-1676264447903 2023-02-13 05:00:48,456 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb0-1676264436786-1676264447903" 2023-02-13 05:00:48,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb0-1676264436786-1676264447903 2023-02-13 05:00:48,460 INFO [Listener at localhost/42165] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676264436786 2023-02-13 05:00:48,461 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676264436786 2023-02-13 05:00:48,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=92, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:48,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-13 05:00:48,465 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676264436786 2023-02-13 05:00:48,465 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264448465"}]},"ts":"1676264448465"} 2023-02-13 05:00:48,466 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=ENABLING in hbase:meta 2023-02-13 05:00:48,469 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 05:00:48,471 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN}, {pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN}, {pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN}, {pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN}, {pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN}, {pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN}] 2023-02-13 05:00:48,472 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN 2023-02-13 05:00:48,473 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN 2023-02-13 05:00:48,473 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN 2023-02-13 05:00:48,473 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN 2023-02-13 05:00:48,473 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN 2023-02-13 05:00:48,474 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=98, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 05:00:48,474 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=94, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 05:00:48,474 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=97, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 05:00:48,474 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=93, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 05:00:48,474 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=96, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 05:00:48,474 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN 2023-02-13 05:00:48,475 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=95, ppid=92, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 05:00:48,624 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 05:00:48,624 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 05:00:48,624 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 05:00:48,624 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 05:00:48,624 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 05:00:48,628 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:48,628 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:48,628 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:48,628 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:48,628 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264448627"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264448627"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264448627"}]},"ts":"1676264448627"} 2023-02-13 05:00:48,628 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264448627"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264448627"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264448627"}]},"ts":"1676264448627"} 2023-02-13 05:00:48,628 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:48,628 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264448627"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264448627"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264448627"}]},"ts":"1676264448627"} 2023-02-13 05:00:48,628 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264448627"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264448627"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264448627"}]},"ts":"1676264448627"} 2023-02-13 05:00:48,628 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264448627"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264448627"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264448627"}]},"ts":"1676264448627"} 2023-02-13 05:00:48,630 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=99, ppid=97, state=RUNNABLE; OpenRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:48,631 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=100, ppid=96, state=RUNNABLE; OpenRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:48,631 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=101, ppid=95, state=RUNNABLE; OpenRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:48,633 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=102, ppid=98, state=RUNNABLE; OpenRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:48,633 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:48,633 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264448633"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264448633"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264448633"}]},"ts":"1676264448633"} 2023-02-13 05:00:48,634 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=103, ppid=93, state=RUNNABLE; OpenRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:48,639 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=104, ppid=94, state=RUNNABLE; OpenRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:48,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-13 05:00:48,784 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:48,784 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:48,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d429d785f30801586e3db3ae3a9428b8, NAME => 'testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 05:00:48,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ee2d32fb6918e03288e862be581d9bcb, NAME => 'testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 05:00:48,785 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:48,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7017e1f9205c2ebde9f23df4733114d5, NAME => 'testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 05:00:48,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:48,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:48,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:48,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:48,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:48,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:48,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:48,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:48,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:48,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:48,786 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:48,786 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:48,787 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:48,787 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:48,787 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:48,787 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf 2023-02-13 05:00:48,788 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf 2023-02-13 05:00:48,788 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf 2023-02-13 05:00:48,788 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf 2023-02-13 05:00:48,788 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf 2023-02-13 05:00:48,788 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf 2023-02-13 05:00:48,788 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d429d785f30801586e3db3ae3a9428b8 columnFamilyName cf 2023-02-13 05:00:48,788 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee2d32fb6918e03288e862be581d9bcb columnFamilyName cf 2023-02-13 05:00:48,788 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7017e1f9205c2ebde9f23df4733114d5 columnFamilyName cf 2023-02-13 05:00:48,793 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/d3866b21c6944cf0882c65fec182cb40 2023-02-13 05:00:48,793 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/811ac31223674808858dc07f9153be03 2023-02-13 05:00:48,793 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(310): Store=ee2d32fb6918e03288e862be581d9bcb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:48,793 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(310): Store=d429d785f30801586e3db3ae3a9428b8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:48,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:48,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:48,795 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d15ef8effb63426f88cf28b863b418da 2023-02-13 05:00:48,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:48,795 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(310): Store=7017e1f9205c2ebde9f23df4733114d5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:48,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:48,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:48,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:48,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:48,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:48,799 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d429d785f30801586e3db3ae3a9428b8; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73327143, jitterRate=0.09265957772731781}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:48,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d429d785f30801586e3db3ae3a9428b8: Running coprocessor pre-open hook at 1676264448786Writing region info on filesystem at 1676264448786Initializing all the Stores at 1676264448786Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264448786Cleaning up temporary data from old regions at 1676264448796 (+10 ms)Cleaning up detritus from prior splits at 1676264448797 (+1 ms)Running coprocessor post-open hooks at 1676264448799 (+2 ms)Region opened successfully at 1676264448799 2023-02-13 05:00:48,800 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ee2d32fb6918e03288e862be581d9bcb; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67832048, jitterRate=0.010776281356811523}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:48,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ee2d32fb6918e03288e862be581d9bcb: Running coprocessor pre-open hook at 1676264448786Writing region info on filesystem at 1676264448786Initializing all the Stores at 1676264448786Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264448786Cleaning up temporary data from old regions at 1676264448796 (+10 ms)Cleaning up detritus from prior splits at 1676264448797 (+1 ms)Running coprocessor post-open hooks at 1676264448800 (+3 ms)Region opened successfully at 1676264448800 2023-02-13 05:00:48,800 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8., pid=103, masterSystemTime=1676264448782 2023-02-13 05:00:48,800 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:48,801 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb., pid=99, masterSystemTime=1676264448782 2023-02-13 05:00:48,801 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7017e1f9205c2ebde9f23df4733114d5; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63800924, jitterRate=-0.049292147159576416}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:48,801 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7017e1f9205c2ebde9f23df4733114d5: Running coprocessor pre-open hook at 1676264448786Writing region info on filesystem at 1676264448786Initializing all the Stores at 1676264448786Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264448786Cleaning up temporary data from old regions at 1676264448798 (+12 ms)Cleaning up detritus from prior splits at 1676264448799 (+1 ms)Running coprocessor post-open hooks at 1676264448801 (+2 ms)Region opened successfully at 1676264448801 2023-02-13 05:00:48,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:48,802 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:48,802 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:48,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 347ea8719201fbf1ee1e731b16986087, NAME => 'testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 05:00:48,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:48,802 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5., pid=102, masterSystemTime=1676264448783 2023-02-13 05:00:48,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:48,802 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:48,802 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=93 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:48,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:48,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:48,803 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264448802"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264448802"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264448802"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264448802"}]},"ts":"1676264448802"} 2023-02-13 05:00:48,803 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:48,803 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:48,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d063b19c8c76cc04b197243b846f83ab, NAME => 'testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 05:00:48,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:48,803 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:48,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:48,803 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=97 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:48,804 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264448803"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264448803"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264448803"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264448803"}]},"ts":"1676264448803"} 2023-02-13 05:00:48,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:48,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:48,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:48,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:48,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 11ed57e84fcc1f7d47e9ad2ed09021f8, NAME => 'testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 05:00:48,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:48,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:48,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:48,805 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=98 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:48,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:48,805 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:48,805 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:48,805 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264448805"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264448805"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264448805"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264448805"}]},"ts":"1676264448805"} 2023-02-13 05:00:48,806 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf 2023-02-13 05:00:48,806 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf 2023-02-13 05:00:48,807 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 347ea8719201fbf1ee1e731b16986087 columnFamilyName cf 2023-02-13 05:00:48,807 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf 2023-02-13 05:00:48,807 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf 2023-02-13 05:00:48,807 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d063b19c8c76cc04b197243b846f83ab columnFamilyName cf 2023-02-13 05:00:48,808 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=103, resume processing ppid=93 2023-02-13 05:00:48,808 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=103, ppid=93, state=SUCCESS; OpenRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,42589,1676264434015 in 171 msec 2023-02-13 05:00:48,810 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=99, resume processing ppid=97 2023-02-13 05:00:48,810 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=99, ppid=97, state=SUCCESS; OpenRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,34363,1676264434089 in 177 msec 2023-02-13 05:00:48,810 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=102, resume processing ppid=98 2023-02-13 05:00:48,810 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=102, ppid=98, state=SUCCESS; OpenRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,39885,1676264434059 in 175 msec 2023-02-13 05:00:48,810 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=93, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN in 339 msec 2023-02-13 05:00:48,811 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=97, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN in 339 msec 2023-02-13 05:00:48,811 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=98, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN in 339 msec 2023-02-13 05:00:48,816 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:48,817 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf 2023-02-13 05:00:48,817 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf 2023-02-13 05:00:48,817 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 11ed57e84fcc1f7d47e9ad2ed09021f8 columnFamilyName cf 2023-02-13 05:00:48,819 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/5ba29d67350f40888be6bfe3fec7f441 2023-02-13 05:00:48,819 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/519efdc2aee8432c80befe61da0da81c 2023-02-13 05:00:48,819 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(310): Store=d063b19c8c76cc04b197243b846f83ab/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:48,819 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(310): Store=347ea8719201fbf1ee1e731b16986087/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:48,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:48,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:48,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:48,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:48,823 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/6414549fb99340299c9cb4b42e13bd55 2023-02-13 05:00:48,823 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(310): Store=11ed57e84fcc1f7d47e9ad2ed09021f8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:48,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:48,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:48,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:48,825 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 347ea8719201fbf1ee1e731b16986087; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67512083, jitterRate=0.006008431315422058}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:48,825 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 347ea8719201fbf1ee1e731b16986087: Running coprocessor pre-open hook at 1676264448803Writing region info on filesystem at 1676264448803Initializing all the Stores at 1676264448804 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264448804Cleaning up temporary data from old regions at 1676264448821 (+17 ms)Cleaning up detritus from prior splits at 1676264448822 (+1 ms)Running coprocessor post-open hooks at 1676264448825 (+3 ms)Region opened successfully at 1676264448825 2023-02-13 05:00:48,825 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d063b19c8c76cc04b197243b846f83ab; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68967643, jitterRate=0.027697965502738953}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:48,825 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d063b19c8c76cc04b197243b846f83ab: Running coprocessor pre-open hook at 1676264448804Writing region info on filesystem at 1676264448804Initializing all the Stores at 1676264448805 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264448805Cleaning up temporary data from old regions at 1676264448821 (+16 ms)Cleaning up detritus from prior splits at 1676264448822 (+1 ms)Running coprocessor post-open hooks at 1676264448825 (+3 ms)Region opened successfully at 1676264448825 2023-02-13 05:00:48,825 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087., pid=100, masterSystemTime=1676264448782 2023-02-13 05:00:48,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:48,826 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab., pid=104, masterSystemTime=1676264448782 2023-02-13 05:00:48,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:48,827 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:48,828 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=96 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:48,828 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264448828"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264448828"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264448828"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264448828"}]},"ts":"1676264448828"} 2023-02-13 05:00:48,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:48,828 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:48,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:48,830 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 11ed57e84fcc1f7d47e9ad2ed09021f8; next sequenceid=19; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70151936, jitterRate=0.045345306396484375}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:48,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 11ed57e84fcc1f7d47e9ad2ed09021f8: Running coprocessor pre-open hook at 1676264448805Writing region info on filesystem at 1676264448805Initializing all the Stores at 1676264448807 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264448807Cleaning up temporary data from old regions at 1676264448826 (+19 ms)Cleaning up detritus from prior splits at 1676264448827 (+1 ms)Running coprocessor post-open hooks at 1676264448830 (+3 ms)Region opened successfully at 1676264448830 2023-02-13 05:00:48,831 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=94 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:48,831 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264448831"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264448831"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264448831"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264448831"}]},"ts":"1676264448831"} 2023-02-13 05:00:48,831 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8., pid=101, masterSystemTime=1676264448783 2023-02-13 05:00:48,833 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=100, resume processing ppid=96 2023-02-13 05:00:48,833 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=100, ppid=96, state=SUCCESS; OpenRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,42589,1676264434015 in 199 msec 2023-02-13 05:00:48,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:48,833 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:48,834 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=95 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=OPEN, openSeqNum=19, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:48,834 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264448833"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264448833"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264448833"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264448833"}]},"ts":"1676264448833"} 2023-02-13 05:00:48,834 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=96, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN in 362 msec 2023-02-13 05:00:48,835 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=104, resume processing ppid=94 2023-02-13 05:00:48,835 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=104, ppid=94, state=SUCCESS; OpenRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,34363,1676264434089 in 194 msec 2023-02-13 05:00:48,836 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=94, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN in 366 msec 2023-02-13 05:00:48,837 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=101, resume processing ppid=95 2023-02-13 05:00:48,837 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=101, ppid=95, state=SUCCESS; OpenRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,39885,1676264434059 in 204 msec 2023-02-13 05:00:48,838 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=95, resume processing ppid=92 2023-02-13 05:00:48,838 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=95, ppid=92, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN in 366 msec 2023-02-13 05:00:48,839 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264448839"}]},"ts":"1676264448839"} 2023-02-13 05:00:48,840 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=ENABLED in hbase:meta 2023-02-13 05:00:48,841 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676264436786' was successfully enabled. 2023-02-13 05:00:48,844 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=92, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676264436786 in 381 msec 2023-02-13 05:00:49,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=92 2023-02-13 05:00:49,216 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676264436786, procId: 92 completed 2023-02-13 05:00:49,217 DEBUG [Listener at localhost/42165] zookeeper.ReadOnlyZKClient(139): Connect 0x3b0f3f9f to 127.0.0.1:63943 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 05:00:49,222 DEBUG [Listener at localhost/42165] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56c31b3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 05:00:49,224 DEBUG [hconnection-0x74f6b663-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:49,225 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44546, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:49,230 DEBUG [hconnection-0x74f6b663-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:49,232 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58212, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:49,341 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 05:00:49,344 DEBUG [hconnection-0x74f6b663-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:49,347 INFO [RS-EventLoopGroup-5-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41024, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:49,432 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 05:00:49,525 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 05:00:49,646 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 05:00:49,738 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 05:00:50,668 INFO [Listener at localhost/42165] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676264436786 2023-02-13 05:00:50,668 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676264436786 2023-02-13 05:00:50,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=105, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:50,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-13 05:00:50,672 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264450672"}]},"ts":"1676264450672"} 2023-02-13 05:00:50,673 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=DISABLING in hbase:meta 2023-02-13 05:00:50,675 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676264436786 to state=DISABLING 2023-02-13 05:00:50,676 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN}, {pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN}, {pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN}, {pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN}, {pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN}, {pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN}] 2023-02-13 05:00:50,677 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=106, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN 2023-02-13 05:00:50,678 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=107, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN 2023-02-13 05:00:50,678 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=110, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN 2023-02-13 05:00:50,678 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=111, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN 2023-02-13 05:00:50,678 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=109, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN 2023-02-13 05:00:50,678 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:50,679 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:50,679 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264450678"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264450678"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264450678"}]},"ts":"1676264450678"} 2023-02-13 05:00:50,679 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264450679"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264450679"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264450679"}]},"ts":"1676264450679"} 2023-02-13 05:00:50,679 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:50,679 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264450679"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264450679"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264450679"}]},"ts":"1676264450679"} 2023-02-13 05:00:50,679 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:50,679 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264450679"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264450679"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264450679"}]},"ts":"1676264450679"} 2023-02-13 05:00:50,680 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=112, ppid=106, state=RUNNABLE; CloseRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:50,680 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:50,681 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264450680"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264450680"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264450680"}]},"ts":"1676264450680"} 2023-02-13 05:00:50,683 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=113, ppid=107, state=RUNNABLE; CloseRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:50,684 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=114, ppid=110, state=RUNNABLE; CloseRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:50,685 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=115, ppid=111, state=RUNNABLE; CloseRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:50,686 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=116, ppid=109, state=RUNNABLE; CloseRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:50,686 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=108, ppid=105, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN 2023-02-13 05:00:50,687 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:50,687 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264450687"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264450687"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264450687"}]},"ts":"1676264450687"} 2023-02-13 05:00:50,688 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=117, ppid=108, state=RUNNABLE; CloseRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:50,835 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:50,836 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d429d785f30801586e3db3ae3a9428b8, disabling compactions & flushes 2023-02-13 05:00:50,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:50,836 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:50,836 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. after waiting 0 ms 2023-02-13 05:00:50,836 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:50,836 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:50,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d063b19c8c76cc04b197243b846f83ab, disabling compactions & flushes 2023-02-13 05:00:50,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:50,840 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:50,840 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. after waiting 0 ms 2023-02-13 05:00:50,840 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:50,840 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:50,841 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7017e1f9205c2ebde9f23df4733114d5, disabling compactions & flushes 2023-02-13 05:00:50,841 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:50,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:50,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. after waiting 0 ms 2023-02-13 05:00:50,845 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:50,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 05:00:50,848 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 05:00:50,848 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:50,849 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d429d785f30801586e3db3ae3a9428b8: Waiting for close lock at 1676264450835Running coprocessor pre-close hooks at 1676264450835Disabling compacts and flushes for region at 1676264450835Disabling writes for close at 1676264450836 (+1 ms)Writing region close event to WAL at 1676264450840 (+4 ms)Running coprocessor post-close hooks at 1676264450848 (+8 ms)Closed at 1676264450848 2023-02-13 05:00:50,849 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 05:00:50,849 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:50,849 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d063b19c8c76cc04b197243b846f83ab: Waiting for close lock at 1676264450836Running coprocessor pre-close hooks at 1676264450836Disabling compacts and flushes for region at 1676264450836Disabling writes for close at 1676264450840 (+4 ms)Writing region close event to WAL at 1676264450845 (+5 ms)Running coprocessor post-close hooks at 1676264450849 (+4 ms)Closed at 1676264450849 2023-02-13 05:00:50,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:50,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7017e1f9205c2ebde9f23df4733114d5: Waiting for close lock at 1676264450841Running coprocessor pre-close hooks at 1676264450841Disabling compacts and flushes for region at 1676264450841Disabling writes for close at 1676264450845 (+4 ms)Writing region close event to WAL at 1676264450846 (+1 ms)Running coprocessor post-close hooks at 1676264450849 (+3 ms)Closed at 1676264450849 2023-02-13 05:00:50,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:50,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:50,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 347ea8719201fbf1ee1e731b16986087, disabling compactions & flushes 2023-02-13 05:00:50,850 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:50,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:50,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. after waiting 0 ms 2023-02-13 05:00:50,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:50,851 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=106 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=CLOSED 2023-02-13 05:00:50,851 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264450851"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264450851"}]},"ts":"1676264450851"} 2023-02-13 05:00:50,851 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:50,851 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:50,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ee2d32fb6918e03288e862be581d9bcb, disabling compactions & flushes 2023-02-13 05:00:50,852 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:50,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:50,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. after waiting 0 ms 2023-02-13 05:00:50,852 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:50,853 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=107 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=CLOSED 2023-02-13 05:00:50,853 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264450853"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264450853"}]},"ts":"1676264450853"} 2023-02-13 05:00:50,853 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:50,853 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:50,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 11ed57e84fcc1f7d47e9ad2ed09021f8, disabling compactions & flushes 2023-02-13 05:00:50,853 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:50,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:50,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. after waiting 0 ms 2023-02-13 05:00:50,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:50,854 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=111 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=CLOSED 2023-02-13 05:00:50,854 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264450854"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264450854"}]},"ts":"1676264450854"} 2023-02-13 05:00:50,855 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 05:00:50,856 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:50,856 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 347ea8719201fbf1ee1e731b16986087: Waiting for close lock at 1676264450850Running coprocessor pre-close hooks at 1676264450850Disabling compacts and flushes for region at 1676264450850Disabling writes for close at 1676264450850Writing region close event to WAL at 1676264450851 (+1 ms)Running coprocessor post-close hooks at 1676264450856 (+5 ms)Closed at 1676264450856 2023-02-13 05:00:50,856 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=112, resume processing ppid=106 2023-02-13 05:00:50,856 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=112, ppid=106, state=SUCCESS; CloseRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,42589,1676264434015 in 173 msec 2023-02-13 05:00:50,856 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 05:00:50,857 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:50,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ee2d32fb6918e03288e862be581d9bcb: Waiting for close lock at 1676264450851Running coprocessor pre-close hooks at 1676264450851Disabling compacts and flushes for region at 1676264450851Disabling writes for close at 1676264450852 (+1 ms)Writing region close event to WAL at 1676264450852Running coprocessor post-close hooks at 1676264450857 (+5 ms)Closed at 1676264450857 2023-02-13 05:00:50,858 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=106, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN in 180 msec 2023-02-13 05:00:50,858 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:50,859 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=113, resume processing ppid=107 2023-02-13 05:00:50,859 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=109 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=CLOSED 2023-02-13 05:00:50,859 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=115, resume processing ppid=111 2023-02-13 05:00:50,859 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=113, ppid=107, state=SUCCESS; CloseRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,34363,1676264434089 in 174 msec 2023-02-13 05:00:50,860 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=115, ppid=111, state=SUCCESS; CloseRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,39885,1676264434059 in 171 msec 2023-02-13 05:00:50,860 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264450859"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264450859"}]},"ts":"1676264450859"} 2023-02-13 05:00:50,860 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/recovered.edits/21.seqid, newMaxSeqId=21, maxSeqId=18 2023-02-13 05:00:50,860 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:50,860 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=110 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=CLOSED 2023-02-13 05:00:50,860 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264450860"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264450860"}]},"ts":"1676264450860"} 2023-02-13 05:00:50,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:50,861 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 11ed57e84fcc1f7d47e9ad2ed09021f8: Waiting for close lock at 1676264450853Running coprocessor pre-close hooks at 1676264450853Disabling compacts and flushes for region at 1676264450853Disabling writes for close at 1676264450854 (+1 ms)Writing region close event to WAL at 1676264450854Running coprocessor post-close hooks at 1676264450860 (+6 ms)Closed at 1676264450861 (+1 ms) 2023-02-13 05:00:50,861 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=111, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN in 184 msec 2023-02-13 05:00:50,861 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=107, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN in 184 msec 2023-02-13 05:00:50,862 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:50,863 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=108 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=CLOSED 2023-02-13 05:00:50,863 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264450863"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264450863"}]},"ts":"1676264450863"} 2023-02-13 05:00:50,863 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=116, resume processing ppid=109 2023-02-13 05:00:50,863 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=116, ppid=109, state=SUCCESS; CloseRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,42589,1676264434015 in 175 msec 2023-02-13 05:00:50,864 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=114, resume processing ppid=110 2023-02-13 05:00:50,864 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=114, ppid=110, state=SUCCESS; CloseRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,34363,1676264434089 in 178 msec 2023-02-13 05:00:50,864 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=109, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN in 187 msec 2023-02-13 05:00:50,865 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=110, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN in 188 msec 2023-02-13 05:00:50,865 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=117, resume processing ppid=108 2023-02-13 05:00:50,865 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=117, ppid=108, state=SUCCESS; CloseRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,39885,1676264434059 in 176 msec 2023-02-13 05:00:50,867 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=108, resume processing ppid=105 2023-02-13 05:00:50,867 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=108, ppid=105, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN in 189 msec 2023-02-13 05:00:50,867 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264450867"}]},"ts":"1676264450867"} 2023-02-13 05:00:50,868 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=DISABLED in hbase:meta 2023-02-13 05:00:50,870 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676264436786 to state=DISABLED 2023-02-13 05:00:50,871 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=105, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676264436786 in 202 msec 2023-02-13 05:00:50,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=105 2023-02-13 05:00:50,923 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676264436786, procId: 105 completed 2023-02-13 05:00:50,931 INFO [Listener at localhost/42165] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676264436786-1676264450931 2023-02-13 05:00:50,932 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-emptySnaptb-1676264436786-1676264450931 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } 2023-02-13 05:00:50,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676264450932 (current time:1676264450932). 2023-02-13 05:00:50,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 05:00:50,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] zookeeper.ReadOnlyZKClient(139): Connect 0x77b1f3de to 127.0.0.1:63943 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 05:00:50,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@37eaa72, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 05:00:50,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:50,941 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44560, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:50,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77b1f3de to 127.0.0.1:63943 2023-02-13 05:00:50,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:00:50,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 05:00:50,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-emptySnaptb-1676264436786-1676264450931 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } 2023-02-13 05:00:50,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE 2023-02-13 05:00:50,945 DEBUG [PEWorker-4] locking.LockProcedure(309): LOCKED pid=118, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE 2023-02-13 05:00:50,946 INFO [PEWorker-4] procedure2.TimeoutExecutorThread(81): ADDED pid=118, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE; timeout=600000, timestamp=1676265050946 2023-02-13 05:00:50,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-emptySnaptb-1676264436786-1676264450931 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } 2023-02-13 05:00:50,946 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676264436786-1676264450931 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676264436786 2023-02-13 05:00:50,946 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-emptySnaptb-1676264436786-1676264450931 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 05:00:50,947 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 05:00:50,956 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676264436786-1676264450931 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } 2023-02-13 05:00:50,960 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:50,960 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:50,960 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:50,960 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:50,962 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:50,962 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:50,963 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:50,962 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:50,963 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:50,963 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:50,963 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:50,963 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:50,963 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:50,963 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:50,964 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/519efdc2aee8432c80befe61da0da81c] hfiles 2023-02-13 05:00:50,964 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/519efdc2aee8432c80befe61da0da81c 2023-02-13 05:00:50,965 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/5ba29d67350f40888be6bfe3fec7f441] hfiles 2023-02-13 05:00:50,965 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/5ba29d67350f40888be6bfe3fec7f441 2023-02-13 05:00:50,965 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d15ef8effb63426f88cf28b863b418da] hfiles 2023-02-13 05:00:50,965 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/811ac31223674808858dc07f9153be03] hfiles 2023-02-13 05:00:50,965 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/811ac31223674808858dc07f9153be03 2023-02-13 05:00:50,965 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d15ef8effb63426f88cf28b863b418da 2023-02-13 05:00:50,966 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/6414549fb99340299c9cb4b42e13bd55] hfiles 2023-02-13 05:00:50,966 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/6414549fb99340299c9cb4b42e13bd55 2023-02-13 05:00:50,969 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2] hfiles 2023-02-13 05:00:50,967 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/d3866b21c6944cf0882c65fec182cb40] hfiles 2023-02-13 05:00:50,970 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 2023-02-13 05:00:50,970 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/d3866b21c6944cf0882c65fec182cb40 2023-02-13 05:00:50,970 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 2023-02-13 05:00:50,970 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 2023-02-13 05:00:50,970 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 2023-02-13 05:00:50,970 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 2023-02-13 05:00:50,970 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 2023-02-13 05:00:51,006 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-emptySnaptb-1676264436786-1676264450931 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } as finished. 2023-02-13 05:00:51,006 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-emptySnaptb-1676264436786-1676264450931 2023-02-13 05:00:51,006 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676264436786-1676264450931 2023-02-13 05:00:51,030 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676264436786-1676264450931 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/hbase-failsafe-emptySnaptb-1676264436786-1676264450931 2023-02-13 05:00:51,050 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-emptySnaptb-1676264436786-1676264450931 of table testRestoreSnapshot-1676264436786 completed 2023-02-13 05:00:51,050 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676264436786-1676264450931 2023-02-13 05:00:51,051 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/hbase-failsafe-emptySnaptb-1676264436786-1676264450931 2023-02-13 05:00:51,051 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-emptySnaptb-1676264436786-1676264450931 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676264436786 at 1676264450946Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-emptySnaptb-1676264436786-1676264450931 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } at 1676264450956 (+10 ms)Consolidate snapshot: hbase-failsafe-emptySnaptb-1676264436786-1676264450931 at 1676264451006 (+50 ms)Loading Region manifests for hbase-failsafe-emptySnaptb-1676264436786-1676264450931 at 1676264451006Writing data manifest for hbase-failsafe-emptySnaptb-1676264436786-1676264450931 at 1676264451013 (+7 ms)Verifying snapshot: hbase-failsafe-emptySnaptb-1676264436786-1676264450931 at 1676264451021 (+8 ms)Snapshot hbase-failsafe-emptySnaptb-1676264436786-1676264450931 of table testRestoreSnapshot-1676264436786 completed at 1676264451050 (+29 ms) 2023-02-13 05:00:51,053 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=118, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE 2023-02-13 05:00:51,055 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=118, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE in 109 msec 2023-02-13 05:00:51,197 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 05:00:51,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-emptySnaptb-1676264436786-1676264450931 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } is done 2023-02-13 05:00:51,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-emptySnaptb-1676264436786-1676264450931 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 05:00:51,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=119, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676264436786 snapshot=name: "emptySnaptb-1676264436786" table: "testRestoreSnapshot-1676264436786" creation_time: 1676264437924 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 05:00:51,203 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(880): Restore snapshot=emptySnaptb-1676264436786 as table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:51,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-13 05:00:51,216 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/.tabledesc/.tableinfo.0000000003 2023-02-13 05:00:51,216 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/.tabledesc/.tableinfo.0000000002 2023-02-13 05:00:51,216 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/.tabledesc/.tableinfo.0000000003 2023-02-13 05:00:51,217 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=emptySnaptb-1676264436786 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } 2023-02-13 05:00:51,218 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "emptySnaptb-1676264436786" table: "testRestoreSnapshot-1676264436786" creation_time: 1676264437924 type: DISABLED version: 2 ttl: 0 2023-02-13 05:00:51,219 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786 2023-02-13 05:00:51,226 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:51,229 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 2023-02-13 05:00:51,230 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 2023-02-13 05:00:51,231 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 2023-02-13 05:00:51,232 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 2023-02-13 05:00:51,233 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 2023-02-13 05:00:51,234 DEBUG [RestoreSnapshot-pool-0] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 2023-02-13 05:00:51,235 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:51,235 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:51,235 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:51,235 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:51,235 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:51,235 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(223): region to restore: ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:51,239 DEBUG [RestoreSnapshot-pool-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/6414549fb99340299c9cb4b42e13bd55 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/6414549fb99340299c9cb4b42e13bd55 2023-02-13 05:00:51,239 DEBUG [RestoreSnapshot-pool-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/519efdc2aee8432c80befe61da0da81c to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/519efdc2aee8432c80befe61da0da81c 2023-02-13 05:00:51,240 DEBUG [RestoreSnapshot-pool-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/811ac31223674808858dc07f9153be03 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/811ac31223674808858dc07f9153be03 2023-02-13 05:00:51,241 DEBUG [RestoreSnapshot-pool-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d15ef8effb63426f88cf28b863b418da to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/d15ef8effb63426f88cf28b863b418da 2023-02-13 05:00:51,241 DEBUG [RestoreSnapshot-pool-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/5ba29d67350f40888be6bfe3fec7f441 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/5ba29d67350f40888be6bfe3fec7f441 2023-02-13 05:00:51,241 DEBUG [RestoreSnapshot-pool-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/d3866b21c6944cf0882c65fec182cb40 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/d3866b21c6944cf0882c65fec182cb40 2023-02-13 05:00:51,242 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "emptySnaptb-1676264436786" table: "testRestoreSnapshot-1676264436786" creation_time: 1676264437924 type: DISABLED version: 2 ttl: 0 2023-02-13 05:00:51,243 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264451243"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:51,243 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264451243"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:51,243 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264451243"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:51,243 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264451243"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:51,243 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264451243"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:51,243 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264451243"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:51,245 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 05:00:51,245 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 11ed57e84fcc1f7d47e9ad2ed09021f8, NAME => 'testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 347ea8719201fbf1ee1e731b16986087, NAME => 'testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 7017e1f9205c2ebde9f23df4733114d5, NAME => 'testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => d063b19c8c76cc04b197243b846f83ab, NAME => 'testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => d429d785f30801586e3db3ae3a9428b8, NAME => 'testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => ee2d32fb6918e03288e862be581d9bcb, NAME => 'testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.', STARTKEY => '4', ENDKEY => '5'}] 2023-02-13 05:00:51,245 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264451244"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264451244"}]},"ts":"1676264451244"} 2023-02-13 05:00:51,245 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264451244"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264451244"}]},"ts":"1676264451244"} 2023-02-13 05:00:51,245 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264451244"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264451244"}]},"ts":"1676264451244"} 2023-02-13 05:00:51,246 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264451244"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264451244"}]},"ts":"1676264451244"} 2023-02-13 05:00:51,246 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264451244"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264451244"}]},"ts":"1676264451244"} 2023-02-13 05:00:51,246 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264451244"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264451244"}]},"ts":"1676264451244"} 2023-02-13 05:00:51,248 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 05:00:51,248 INFO [PEWorker-3] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-13 05:00:51,248 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 11ed57e84fcc1f7d47e9ad2ed09021f8, NAME => 'testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 347ea8719201fbf1ee1e731b16986087, NAME => 'testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 7017e1f9205c2ebde9f23df4733114d5, NAME => 'testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => d063b19c8c76cc04b197243b846f83ab, NAME => 'testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => d429d785f30801586e3db3ae3a9428b8, NAME => 'testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => ee2d32fb6918e03288e862be581d9bcb, NAME => 'testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.', STARTKEY => '4', ENDKEY => '5'}] 2023-02-13 05:00:51,248 INFO [PEWorker-3] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=emptySnaptb-1676264436786 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676264436786 completed! 2023-02-13 05:00:51,250 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=119, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676264436786 snapshot=name: "emptySnaptb-1676264436786" table: "testRestoreSnapshot-1676264436786" creation_time: 1676264437924 type: DISABLED version: 2 ttl: 0 ) in 46 msec 2023-02-13 05:00:51,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=119 2023-02-13 05:00:51,456 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676264436786, procId: 119 completed 2023-02-13 05:00:51,457 INFO [Listener at localhost/42165] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-emptySnaptb-1676264436786-1676264450931 2023-02-13 05:00:51,457 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-emptySnaptb-1676264436786-1676264450931" 2023-02-13 05:00:51,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-emptySnaptb-1676264436786-1676264450931 2023-02-13 05:00:51,460 INFO [Listener at localhost/42165] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676264436786 2023-02-13 05:00:51,461 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676264436786 2023-02-13 05:00:51,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=120, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:51,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-13 05:00:51,465 INFO [PEWorker-1] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676264436786 2023-02-13 05:00:51,465 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264451465"}]},"ts":"1676264451465"} 2023-02-13 05:00:51,467 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=ENABLING in hbase:meta 2023-02-13 05:00:51,469 DEBUG [PEWorker-1] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 05:00:51,470 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN}, {pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN}, {pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN}, {pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN}, {pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN}, {pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN}] 2023-02-13 05:00:51,471 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN 2023-02-13 05:00:51,472 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN 2023-02-13 05:00:51,472 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN 2023-02-13 05:00:51,472 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN 2023-02-13 05:00:51,472 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=121, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 05:00:51,472 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=122, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 05:00:51,473 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=123, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 05:00:51,473 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=124, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 05:00:51,473 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN 2023-02-13 05:00:51,473 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN 2023-02-13 05:00:51,474 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=125, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 05:00:51,474 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=126, ppid=120, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 05:00:51,623 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 05:00:51,623 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 05:00:51,623 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 05:00:51,623 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 05:00:51,623 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 05:00:51,628 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:51,628 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:51,628 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:51,628 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:51,628 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264451628"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264451628"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264451628"}]},"ts":"1676264451628"} 2023-02-13 05:00:51,628 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264451628"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264451628"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264451628"}]},"ts":"1676264451628"} 2023-02-13 05:00:51,628 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:51,628 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264451628"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264451628"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264451628"}]},"ts":"1676264451628"} 2023-02-13 05:00:51,628 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264451628"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264451628"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264451628"}]},"ts":"1676264451628"} 2023-02-13 05:00:51,628 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264451628"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264451628"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264451628"}]},"ts":"1676264451628"} 2023-02-13 05:00:51,630 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=127, ppid=126, state=RUNNABLE; OpenRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:51,631 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=128, ppid=122, state=RUNNABLE; OpenRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:51,632 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=129, ppid=121, state=RUNNABLE; OpenRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:51,633 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=130, ppid=124, state=RUNNABLE; OpenRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:51,633 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:51,633 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264451633"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264451633"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264451633"}]},"ts":"1676264451633"} 2023-02-13 05:00:51,634 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=131, ppid=125, state=RUNNABLE; OpenRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:51,635 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=132, ppid=123, state=RUNNABLE; OpenRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:51,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-13 05:00:51,784 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:51,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 11ed57e84fcc1f7d47e9ad2ed09021f8, NAME => 'testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 05:00:51,784 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:51,785 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:51,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:51,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ee2d32fb6918e03288e862be581d9bcb, NAME => 'testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 05:00:51,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:51,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:51,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:51,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:51,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:51,785 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:51,786 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:51,786 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:51,786 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d429d785f30801586e3db3ae3a9428b8, NAME => 'testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 05:00:51,786 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:51,786 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:51,786 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:51,786 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:51,786 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:51,787 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf 2023-02-13 05:00:51,787 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf 2023-02-13 05:00:51,787 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf 2023-02-13 05:00:51,788 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf 2023-02-13 05:00:51,788 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:51,788 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 11ed57e84fcc1f7d47e9ad2ed09021f8 columnFamilyName cf 2023-02-13 05:00:51,788 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee2d32fb6918e03288e862be581d9bcb columnFamilyName cf 2023-02-13 05:00:51,789 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf 2023-02-13 05:00:51,789 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf 2023-02-13 05:00:51,789 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(310): Store=ee2d32fb6918e03288e862be581d9bcb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:51,789 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(310): Store=11ed57e84fcc1f7d47e9ad2ed09021f8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:51,789 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d429d785f30801586e3db3ae3a9428b8 columnFamilyName cf 2023-02-13 05:00:51,790 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(310): Store=d429d785f30801586e3db3ae3a9428b8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:51,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:51,790 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:51,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:51,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:51,791 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:51,792 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:51,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:51,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:51,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:51,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ee2d32fb6918e03288e862be581d9bcb; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73616538, jitterRate=0.09697189927101135}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:51,794 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 11ed57e84fcc1f7d47e9ad2ed09021f8; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60108435, jitterRate=-0.10431452095508575}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:51,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ee2d32fb6918e03288e862be581d9bcb: Running coprocessor pre-open hook at 1676264451785Writing region info on filesystem at 1676264451785Initializing all the Stores at 1676264451786 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264451786Cleaning up temporary data from old regions at 1676264451792 (+6 ms)Cleaning up detritus from prior splits at 1676264451792Running coprocessor post-open hooks at 1676264451794 (+2 ms)Region opened successfully at 1676264451794 2023-02-13 05:00:51,794 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 11ed57e84fcc1f7d47e9ad2ed09021f8: Running coprocessor pre-open hook at 1676264451785Writing region info on filesystem at 1676264451785Initializing all the Stores at 1676264451785Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264451786 (+1 ms)Cleaning up temporary data from old regions at 1676264451792 (+6 ms)Cleaning up detritus from prior splits at 1676264451792Running coprocessor post-open hooks at 1676264451794 (+2 ms)Region opened successfully at 1676264451794 2023-02-13 05:00:51,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d429d785f30801586e3db3ae3a9428b8; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60490942, jitterRate=-0.09861472249031067}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:51,795 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d429d785f30801586e3db3ae3a9428b8: Running coprocessor pre-open hook at 1676264451787Writing region info on filesystem at 1676264451787Initializing all the Stores at 1676264451787Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264451787Cleaning up temporary data from old regions at 1676264451792 (+5 ms)Cleaning up detritus from prior splits at 1676264451792Running coprocessor post-open hooks at 1676264451795 (+3 ms)Region opened successfully at 1676264451795 2023-02-13 05:00:51,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb., pid=131, masterSystemTime=1676264451783 2023-02-13 05:00:51,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8., pid=132, masterSystemTime=1676264451782 2023-02-13 05:00:51,795 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8., pid=129, masterSystemTime=1676264451784 2023-02-13 05:00:51,796 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:51,797 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:51,797 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:51,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d063b19c8c76cc04b197243b846f83ab, NAME => 'testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 05:00:51,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:51,797 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=125 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:51,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:51,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:51,797 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264451797"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264451797"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264451797"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264451797"}]},"ts":"1676264451797"} 2023-02-13 05:00:51,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:51,797 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:51,798 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:51,798 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=123 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:51,798 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264451797"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264451797"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264451797"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264451797"}]},"ts":"1676264451797"} 2023-02-13 05:00:51,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:51,798 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:51,798 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:51,798 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:51,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 347ea8719201fbf1ee1e731b16986087, NAME => 'testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 05:00:51,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7017e1f9205c2ebde9f23df4733114d5, NAME => 'testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 05:00:51,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:51,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:51,798 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=121 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:51,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:51,799 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:51,798 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:51,799 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264451798"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264451798"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264451798"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264451798"}]},"ts":"1676264451798"} 2023-02-13 05:00:51,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:51,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:51,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:51,799 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:51,800 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:51,800 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:51,800 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf 2023-02-13 05:00:51,801 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf 2023-02-13 05:00:51,801 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d063b19c8c76cc04b197243b846f83ab columnFamilyName cf 2023-02-13 05:00:51,801 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=131, resume processing ppid=125 2023-02-13 05:00:51,801 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=131, ppid=125, state=SUCCESS; OpenRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,42589,1676264434015 in 165 msec 2023-02-13 05:00:51,802 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=132, resume processing ppid=123 2023-02-13 05:00:51,802 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=132, ppid=123, state=SUCCESS; OpenRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,34363,1676264434089 in 165 msec 2023-02-13 05:00:51,803 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf 2023-02-13 05:00:51,803 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf 2023-02-13 05:00:51,803 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(310): Store=d063b19c8c76cc04b197243b846f83ab/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:51,803 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf 2023-02-13 05:00:51,803 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf 2023-02-13 05:00:51,803 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=129, resume processing ppid=121 2023-02-13 05:00:51,803 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=125, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN in 331 msec 2023-02-13 05:00:51,803 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7017e1f9205c2ebde9f23df4733114d5 columnFamilyName cf 2023-02-13 05:00:51,803 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=129, ppid=121, state=SUCCESS; OpenRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059 in 169 msec 2023-02-13 05:00:51,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:51,804 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=123, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN in 332 msec 2023-02-13 05:00:51,803 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 347ea8719201fbf1ee1e731b16986087 columnFamilyName cf 2023-02-13 05:00:51,804 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=121, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN in 333 msec 2023-02-13 05:00:51,805 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(310): Store=7017e1f9205c2ebde9f23df4733114d5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:51,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:51,805 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(310): Store=347ea8719201fbf1ee1e731b16986087/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:51,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:51,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:51,807 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:51,808 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d063b19c8c76cc04b197243b846f83ab; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71949672, jitterRate=0.07213366031646729}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:51,808 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d063b19c8c76cc04b197243b846f83ab: Running coprocessor pre-open hook at 1676264451797Writing region info on filesystem at 1676264451797Initializing all the Stores at 1676264451798 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264451798Cleaning up temporary data from old regions at 1676264451805 (+7 ms)Cleaning up detritus from prior splits at 1676264451806 (+1 ms)Running coprocessor post-open hooks at 1676264451808 (+2 ms)Region opened successfully at 1676264451808 2023-02-13 05:00:51,808 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab., pid=128, masterSystemTime=1676264451783 2023-02-13 05:00:51,809 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:51,810 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 347ea8719201fbf1ee1e731b16986087; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65636704, jitterRate=-0.021936893463134766}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:51,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 347ea8719201fbf1ee1e731b16986087: Running coprocessor pre-open hook at 1676264451799Writing region info on filesystem at 1676264451799Initializing all the Stores at 1676264451800 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264451800Cleaning up temporary data from old regions at 1676264451807 (+7 ms)Cleaning up detritus from prior splits at 1676264451807Running coprocessor post-open hooks at 1676264451810 (+3 ms)Region opened successfully at 1676264451810 2023-02-13 05:00:51,810 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:51,810 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:51,811 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087., pid=130, masterSystemTime=1676264451784 2023-02-13 05:00:51,811 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=122 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:51,811 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264451811"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264451811"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264451811"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264451811"}]},"ts":"1676264451811"} 2023-02-13 05:00:51,812 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:51,812 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:51,812 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=124 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:51,812 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264451812"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264451812"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264451812"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264451812"}]},"ts":"1676264451812"} 2023-02-13 05:00:51,813 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=128, resume processing ppid=122 2023-02-13 05:00:51,814 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=128, ppid=122, state=SUCCESS; OpenRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,42589,1676264434015 in 181 msec 2023-02-13 05:00:51,815 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=122, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN in 343 msec 2023-02-13 05:00:51,815 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=130, resume processing ppid=124 2023-02-13 05:00:51,815 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=130, ppid=124, state=SUCCESS; OpenRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,39885,1676264434059 in 180 msec 2023-02-13 05:00:51,816 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=124, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN in 345 msec 2023-02-13 05:00:51,816 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:51,817 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:51,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:51,820 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7017e1f9205c2ebde9f23df4733114d5; next sequenceid=22; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69669278, jitterRate=0.03815314173698425}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:51,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7017e1f9205c2ebde9f23df4733114d5: Running coprocessor pre-open hook at 1676264451799Writing region info on filesystem at 1676264451799Initializing all the Stores at 1676264451800 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264451800Cleaning up temporary data from old regions at 1676264451817 (+17 ms)Cleaning up detritus from prior splits at 1676264451818 (+1 ms)Running coprocessor post-open hooks at 1676264451820 (+2 ms)Region opened successfully at 1676264451820 2023-02-13 05:00:51,820 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5., pid=127, masterSystemTime=1676264451782 2023-02-13 05:00:51,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:51,821 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:51,821 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=126 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=OPEN, openSeqNum=22, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:51,822 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264451821"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264451821"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264451821"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264451821"}]},"ts":"1676264451821"} 2023-02-13 05:00:51,824 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=127, resume processing ppid=126 2023-02-13 05:00:51,824 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=127, ppid=126, state=SUCCESS; OpenRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,34363,1676264434089 in 193 msec 2023-02-13 05:00:51,825 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=126, resume processing ppid=120 2023-02-13 05:00:51,825 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=126, ppid=120, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN in 354 msec 2023-02-13 05:00:51,826 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264451826"}]},"ts":"1676264451826"} 2023-02-13 05:00:51,827 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=ENABLED in hbase:meta 2023-02-13 05:00:51,828 INFO [PEWorker-4] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676264436786' was successfully enabled. 2023-02-13 05:00:51,830 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=120, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676264436786 in 368 msec 2023-02-13 05:00:52,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=120 2023-02-13 05:00:52,217 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676264436786, procId: 120 completed 2023-02-13 05:00:52,217 DEBUG [Listener at localhost/42165] zookeeper.ReadOnlyZKClient(139): Connect 0x05da8053 to 127.0.0.1:63943 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 05:00:52,223 DEBUG [Listener at localhost/42165] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3bfc5430, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 05:00:52,225 DEBUG [hconnection-0x1113c564-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:52,227 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44576, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:52,231 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 05:00:52,233 DEBUG [hconnection-0x1113c564-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:52,235 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58228, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:52,236 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 05:00:52,238 DEBUG [hconnection-0x1113c564-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:52,242 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41036, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:52,242 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 05:00:52,245 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 05:00:52,247 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 05:00:52,252 INFO [Listener at localhost/42165] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,253 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=133, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-13 05:00:52,256 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264452256"}]},"ts":"1676264452256"} 2023-02-13 05:00:52,257 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=DISABLING in hbase:meta 2023-02-13 05:00:52,264 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676264436786 to state=DISABLING 2023-02-13 05:00:52,265 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN}, {pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN}, {pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN}, {pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN}, {pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN}, {pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN}] 2023-02-13 05:00:52,266 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=135, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN 2023-02-13 05:00:52,266 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=134, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN 2023-02-13 05:00:52,266 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=138, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN 2023-02-13 05:00:52,266 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=136, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN 2023-02-13 05:00:52,266 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=139, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN 2023-02-13 05:00:52,267 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:52,267 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264452267"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264452267"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264452267"}]},"ts":"1676264452267"} 2023-02-13 05:00:52,267 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:52,267 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:52,267 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:52,267 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264452267"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264452267"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264452267"}]},"ts":"1676264452267"} 2023-02-13 05:00:52,267 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:52,267 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264452267"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264452267"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264452267"}]},"ts":"1676264452267"} 2023-02-13 05:00:52,267 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264452267"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264452267"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264452267"}]},"ts":"1676264452267"} 2023-02-13 05:00:52,267 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264452267"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264452267"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264452267"}]},"ts":"1676264452267"} 2023-02-13 05:00:52,268 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=140, ppid=135, state=RUNNABLE; CloseRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:52,269 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=141, ppid=138, state=RUNNABLE; CloseRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:52,270 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=142, ppid=134, state=RUNNABLE; CloseRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:52,271 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=143, ppid=136, state=RUNNABLE; CloseRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:52,272 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=137, ppid=133, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN 2023-02-13 05:00:52,272 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=144, ppid=139, state=RUNNABLE; CloseRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:52,273 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:52,273 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264452273"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264452273"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264452273"}]},"ts":"1676264452273"} 2023-02-13 05:00:52,274 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=145, ppid=137, state=RUNNABLE; CloseRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:52,369 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 05:00:52,420 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:52,421 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ee2d32fb6918e03288e862be581d9bcb, disabling compactions & flushes 2023-02-13 05:00:52,422 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:52,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:52,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. after waiting 0 ms 2023-02-13 05:00:52,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:52,422 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:52,422 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:52,423 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d429d785f30801586e3db3ae3a9428b8, disabling compactions & flushes 2023-02-13 05:00:52,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7017e1f9205c2ebde9f23df4733114d5, disabling compactions & flushes 2023-02-13 05:00:52,424 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:52,424 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:52,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:52,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:52,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. after waiting 0 ms 2023-02-13 05:00:52,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. after waiting 0 ms 2023-02-13 05:00:52,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:52,424 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:52,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 05:00:52,428 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:52,428 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ee2d32fb6918e03288e862be581d9bcb: Waiting for close lock at 1676264452421Running coprocessor pre-close hooks at 1676264452421Disabling compacts and flushes for region at 1676264452421Disabling writes for close at 1676264452422 (+1 ms)Writing region close event to WAL at 1676264452422Running coprocessor post-close hooks at 1676264452427 (+5 ms)Closed at 1676264452428 (+1 ms) 2023-02-13 05:00:52,428 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 05:00:52,428 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 05:00:52,429 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:52,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d429d785f30801586e3db3ae3a9428b8: Waiting for close lock at 1676264452423Running coprocessor pre-close hooks at 1676264452423Disabling compacts and flushes for region at 1676264452423Disabling writes for close at 1676264452424 (+1 ms)Writing region close event to WAL at 1676264452425 (+1 ms)Running coprocessor post-close hooks at 1676264452429 (+4 ms)Closed at 1676264452429 2023-02-13 05:00:52,430 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:52,430 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7017e1f9205c2ebde9f23df4733114d5: Waiting for close lock at 1676264452423Running coprocessor pre-close hooks at 1676264452423Disabling compacts and flushes for region at 1676264452423Disabling writes for close at 1676264452424 (+1 ms)Writing region close event to WAL at 1676264452425 (+1 ms)Running coprocessor post-close hooks at 1676264452430 (+5 ms)Closed at 1676264452430 2023-02-13 05:00:52,431 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:52,431 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:52,432 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d063b19c8c76cc04b197243b846f83ab, disabling compactions & flushes 2023-02-13 05:00:52,432 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:52,432 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:52,432 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. after waiting 0 ms 2023-02-13 05:00:52,432 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:52,432 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=138 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=CLOSED 2023-02-13 05:00:52,432 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264452432"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264452432"}]},"ts":"1676264452432"} 2023-02-13 05:00:52,433 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:52,433 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:52,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 347ea8719201fbf1ee1e731b16986087, disabling compactions & flushes 2023-02-13 05:00:52,433 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:52,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:52,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. after waiting 0 ms 2023-02-13 05:00:52,433 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:52,434 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=134 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=CLOSED 2023-02-13 05:00:52,434 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264452434"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264452434"}]},"ts":"1676264452434"} 2023-02-13 05:00:52,435 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:52,435 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:52,436 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=139 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=CLOSED 2023-02-13 05:00:52,436 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264452436"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264452436"}]},"ts":"1676264452436"} 2023-02-13 05:00:52,436 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 05:00:52,437 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 11ed57e84fcc1f7d47e9ad2ed09021f8, disabling compactions & flushes 2023-02-13 05:00:52,437 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:52,437 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:52,437 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. after waiting 0 ms 2023-02-13 05:00:52,437 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:52,438 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:52,438 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d063b19c8c76cc04b197243b846f83ab: Waiting for close lock at 1676264452431Running coprocessor pre-close hooks at 1676264452431Disabling compacts and flushes for region at 1676264452431Disabling writes for close at 1676264452432 (+1 ms)Writing region close event to WAL at 1676264452432Running coprocessor post-close hooks at 1676264452438 (+6 ms)Closed at 1676264452438 2023-02-13 05:00:52,440 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 05:00:52,440 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=141, resume processing ppid=138 2023-02-13 05:00:52,440 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=141, ppid=138, state=SUCCESS; CloseRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,42589,1676264434015 in 166 msec 2023-02-13 05:00:52,440 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:52,441 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:52,441 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 347ea8719201fbf1ee1e731b16986087: Waiting for close lock at 1676264452433Running coprocessor pre-close hooks at 1676264452433Disabling compacts and flushes for region at 1676264452433Disabling writes for close at 1676264452433Writing region close event to WAL at 1676264452435 (+2 ms)Running coprocessor post-close hooks at 1676264452441 (+6 ms)Closed at 1676264452441 2023-02-13 05:00:52,442 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=142, resume processing ppid=134 2023-02-13 05:00:52,442 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=142, ppid=134, state=SUCCESS; CloseRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059 in 167 msec 2023-02-13 05:00:52,442 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=135 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=CLOSED 2023-02-13 05:00:52,442 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264452442"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264452442"}]},"ts":"1676264452442"} 2023-02-13 05:00:52,443 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=144, resume processing ppid=139 2023-02-13 05:00:52,443 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=144, ppid=139, state=SUCCESS; CloseRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,34363,1676264434089 in 167 msec 2023-02-13 05:00:52,443 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:52,443 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=138, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN in 176 msec 2023-02-13 05:00:52,444 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=134, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN in 178 msec 2023-02-13 05:00:52,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/recovered.edits/24.seqid, newMaxSeqId=24, maxSeqId=21 2023-02-13 05:00:52,445 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:52,445 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 11ed57e84fcc1f7d47e9ad2ed09021f8: Waiting for close lock at 1676264452435Running coprocessor pre-close hooks at 1676264452435Disabling compacts and flushes for region at 1676264452435Disabling writes for close at 1676264452437 (+2 ms)Writing region close event to WAL at 1676264452440 (+3 ms)Running coprocessor post-close hooks at 1676264452445 (+5 ms)Closed at 1676264452445 2023-02-13 05:00:52,446 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=137 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=CLOSED 2023-02-13 05:00:52,446 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=139, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN in 179 msec 2023-02-13 05:00:52,446 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264452446"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264452446"}]},"ts":"1676264452446"} 2023-02-13 05:00:52,446 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:52,447 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=140, resume processing ppid=135 2023-02-13 05:00:52,447 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=140, ppid=135, state=SUCCESS; CloseRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,42589,1676264434015 in 176 msec 2023-02-13 05:00:52,447 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=136 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=CLOSED 2023-02-13 05:00:52,447 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264452447"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264452447"}]},"ts":"1676264452447"} 2023-02-13 05:00:52,448 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=135, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN in 183 msec 2023-02-13 05:00:52,449 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=145, resume processing ppid=137 2023-02-13 05:00:52,449 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=145, ppid=137, state=SUCCESS; CloseRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,39885,1676264434059 in 173 msec 2023-02-13 05:00:52,450 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=143, resume processing ppid=136 2023-02-13 05:00:52,450 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=143, ppid=136, state=SUCCESS; CloseRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,34363,1676264434089 in 177 msec 2023-02-13 05:00:52,450 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=137, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN in 185 msec 2023-02-13 05:00:52,451 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=136, resume processing ppid=133 2023-02-13 05:00:52,451 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=136, ppid=133, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN in 186 msec 2023-02-13 05:00:52,452 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264452452"}]},"ts":"1676264452452"} 2023-02-13 05:00:52,453 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=DISABLED in hbase:meta 2023-02-13 05:00:52,454 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676264436786 to state=DISABLED 2023-02-13 05:00:52,456 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=133, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676264436786 in 202 msec 2023-02-13 05:00:52,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=133 2023-02-13 05:00:52,507 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676264436786, procId: 133 completed 2023-02-13 05:00:52,516 INFO [Listener at localhost/42165] client.HBaseAdmin(2586): Taking restore-failsafe snapshot: hbase-failsafe-snaptb1-1676264436786-1676264452516 2023-02-13 05:00:52,517 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=hbase-failsafe-snaptb1-1676264436786-1676264452516 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } 2023-02-13 05:00:52,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676264452517 (current time:1676264452517). 2023-02-13 05:00:52,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 05:00:52,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] zookeeper.ReadOnlyZKClient(139): Connect 0x02e085e7 to 127.0.0.1:63943 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 05:00:52,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d782270, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 05:00:52,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:52,525 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44586, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:52,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02e085e7 to 127.0.0.1:63943 2023-02-13 05:00:52,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:00:52,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 05:00:52,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=hbase-failsafe-snaptb1-1676264436786-1676264452516 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } 2023-02-13 05:00:52,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE 2023-02-13 05:00:52,529 DEBUG [PEWorker-2] locking.LockProcedure(309): LOCKED pid=146, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE 2023-02-13 05:00:52,530 INFO [PEWorker-2] procedure2.TimeoutExecutorThread(81): ADDED pid=146, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE; timeout=600000, timestamp=1676265052530 2023-02-13 05:00:52,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(664): Started snapshot: { ss=hbase-failsafe-snaptb1-1676264436786-1676264452516 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } 2023-02-13 05:00:52,530 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot hbase-failsafe-snaptb1-1676264436786-1676264452516 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,530 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=hbase-failsafe-snaptb1-1676264436786-1676264452516 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 05:00:52,530 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 05:00:52,542 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676264436786-1676264452516 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } 2023-02-13 05:00:52,549 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:52,549 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:52,549 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:52,549 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:52,549 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:52,549 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:52,549 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:52,549 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:52,550 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:52,550 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:52,550 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:52,550 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:52,550 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:00:52,550 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:00:52,552 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:00:52,552 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:00:52,552 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:00:52,552 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:00:52,553 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:00:52,553 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:00:52,554 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:00:52,581 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=hbase-failsafe-snaptb1-1676264436786-1676264452516 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } as finished. 2023-02-13 05:00:52,581 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for hbase-failsafe-snaptb1-1676264436786-1676264452516 2023-02-13 05:00:52,582 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676264436786-1676264452516 2023-02-13 05:00:52,597 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676264436786-1676264452516 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/hbase-failsafe-snaptb1-1676264436786-1676264452516 2023-02-13 05:00:52,613 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot hbase-failsafe-snaptb1-1676264436786-1676264452516 of table testRestoreSnapshot-1676264436786 completed 2023-02-13 05:00:52,613 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676264436786-1676264452516 2023-02-13 05:00:52,613 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/hbase-failsafe-snaptb1-1676264436786-1676264452516 2023-02-13 05:00:52,613 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot hbase-failsafe-snaptb1-1676264436786-1676264452516 C_M_SNAPSHOT_TABLE on table testRestoreSnapshot-1676264436786 at 1676264452530Starting to write region info and WALs for regions for offline snapshot:{ ss=hbase-failsafe-snaptb1-1676264436786-1676264452516 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } at 1676264452542 (+12 ms)Consolidate snapshot: hbase-failsafe-snaptb1-1676264436786-1676264452516 at 1676264452581 (+39 ms)Loading Region manifests for hbase-failsafe-snaptb1-1676264436786-1676264452516 at 1676264452581Writing data manifest for hbase-failsafe-snaptb1-1676264436786-1676264452516 at 1676264452584 (+3 ms)Verifying snapshot: hbase-failsafe-snaptb1-1676264436786-1676264452516 at 1676264452592 (+8 ms)Snapshot hbase-failsafe-snaptb1-1676264436786-1676264452516 of table testRestoreSnapshot-1676264436786 completed at 1676264452613 (+21 ms) 2023-02-13 05:00:52,615 DEBUG [PEWorker-1] locking.LockProcedure(242): UNLOCKED pid=146, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE 2023-02-13 05:00:52,616 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=146, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testRestoreSnapshot-1676264436786, type=EXCLUSIVE in 87 msec 2023-02-13 05:00:52,780 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 05:00:52,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=hbase-failsafe-snaptb1-1676264436786-1676264452516 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 } is done 2023-02-13 05:00:52,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(401): Snapshot '{ ss=hbase-failsafe-snaptb1-1676264436786-1676264452516 table=testRestoreSnapshot-1676264436786 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 05:00:52,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=147, state=RUNNABLE:RESTORE_SNAPSHOT_PRE_OPERATION; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676264436786 snapshot=name: "snaptb1-1676264436786" table: "testRestoreSnapshot-1676264436786" creation_time: 1676264447622 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 05:00:52,796 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(880): Restore snapshot=snaptb1-1676264436786 as table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-13 05:00:52,807 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/.tabledesc/.tableinfo.0000000004 2023-02-13 05:00:52,808 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/.tabledesc/.tableinfo.0000000003 2023-02-13 05:00:52,808 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/.tabledesc/.tableinfo.0000000004 2023-02-13 05:00:52,808 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(374): Starting restore snapshot={ ss=snaptb1-1676264436786 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } 2023-02-13 05:00:52,811 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676264436786" table: "testRestoreSnapshot-1676264436786" creation_time: 1676264447622 type: DISABLED version: 2 ttl: 0 2023-02-13 05:00:52,811 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,817 DEBUG [PEWorker-4] snapshot.RestoreSnapshotHelper(797): found 6 regions for table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,818 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 of snapshot=snaptb1-1676264436786 to region=93bce92e4aef17f80aeb287d234dc9c6 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,821 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 of snapshot=snaptb1-1676264436786 to region=93bce92e4aef17f80aeb287d234dc9c6 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,824 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 of snapshot=snaptb1-1676264436786 to region=93bce92e4aef17f80aeb287d234dc9c6 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,827 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 of snapshot=snaptb1-1676264436786 to region=93bce92e4aef17f80aeb287d234dc9c6 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,830 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c of snapshot=snaptb1-1676264436786 to region=93bce92e4aef17f80aeb287d234dc9c6 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,833 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 of snapshot=snaptb1-1676264436786 to region=93bce92e4aef17f80aeb287d234dc9c6 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,836 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 of snapshot=snaptb1-1676264436786 to region=93bce92e4aef17f80aeb287d234dc9c6 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,840 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 of snapshot=snaptb1-1676264436786 to region=93bce92e4aef17f80aeb287d234dc9c6 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,843 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 of snapshot=snaptb1-1676264436786 to region=93bce92e4aef17f80aeb287d234dc9c6 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,847 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b of snapshot=snaptb1-1676264436786 to region=93bce92e4aef17f80aeb287d234dc9c6 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,851 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 of snapshot=snaptb1-1676264436786 to region=93bce92e4aef17f80aeb287d234dc9c6 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,854 DEBUG [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 of snapshot=snaptb1-1676264436786 to region=93bce92e4aef17f80aeb287d234dc9c6 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,857 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:52,858 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:52,858 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:52,858 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:52,858 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:52,858 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(223): region to restore: ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:52,865 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 3ae4957471034eeb9ed26ec1a046807b of snapshot=snaptb1-1676264436786 to region=11ed57e84fcc1f7d47e9ad2ed09021f8 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,867 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 075fd0e105614d9aaf0995ac7325a2fb of snapshot=snaptb1-1676264436786 to region=d063b19c8c76cc04b197243b846f83ab table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,867 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 3f25ec8511e84234a9c7a722ca070d0b of snapshot=snaptb1-1676264436786 to region=ee2d32fb6918e03288e862be581d9bcb table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,874 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 519efdc2aee8432c80befe61da0da81c of snapshot=snaptb1-1676264436786 to region=347ea8719201fbf1ee1e731b16986087 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,875 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 811ac31223674808858dc07f9153be03 of snapshot=snaptb1-1676264436786 to region=d429d785f30801586e3db3ae3a9428b8 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,875 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d15ef8effb63426f88cf28b863b418da of snapshot=snaptb1-1676264436786 to region=7017e1f9205c2ebde9f23df4733114d5 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,884 DEBUG [RestoreSnapshot-pool-6] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d3866b21c6944cf0882c65fec182cb40 of snapshot=snaptb1-1676264436786 to region=ee2d32fb6918e03288e862be581d9bcb table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,892 DEBUG [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 6414549fb99340299c9cb4b42e13bd55 of snapshot=snaptb1-1676264436786 to region=11ed57e84fcc1f7d47e9ad2ed09021f8 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,896 DEBUG [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink b95f98eee2d241bea094905381835a03 of snapshot=snaptb1-1676264436786 to region=d429d785f30801586e3db3ae3a9428b8 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,901 DEBUG [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink d318209d74c44b3fb2d00363c5890672 of snapshot=snaptb1-1676264436786 to region=7017e1f9205c2ebde9f23df4733114d5 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,901 DEBUG [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink b0b32b9f22144056b97173b2a69002a5 of snapshot=snaptb1-1676264436786 to region=347ea8719201fbf1ee1e731b16986087 table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,902 DEBUG [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(517): Restoring missing HFileLink 5ba29d67350f40888be6bfe3fec7f441 of snapshot=snaptb1-1676264436786 to region=d063b19c8c76cc04b197243b846f83ab table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:52,910 INFO [PEWorker-4] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676264436786" table: "testRestoreSnapshot-1676264436786" creation_time: 1676264447622 type: DISABLED version: 2 ttl: 0 2023-02-13 05:00:52,911 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264452911"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:52,911 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264452911"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:52,911 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264452911"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:52,911 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264452911"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:52,911 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264452911"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:52,912 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264452911"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:52,913 INFO [PEWorker-4] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 05:00:52,914 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 11ed57e84fcc1f7d47e9ad2ed09021f8, NAME => 'testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 347ea8719201fbf1ee1e731b16986087, NAME => 'testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 7017e1f9205c2ebde9f23df4733114d5, NAME => 'testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => d063b19c8c76cc04b197243b846f83ab, NAME => 'testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => d429d785f30801586e3db3ae3a9428b8, NAME => 'testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => ee2d32fb6918e03288e862be581d9bcb, NAME => 'testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.', STARTKEY => '4', ENDKEY => '5'}] 2023-02-13 05:00:52,914 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264452912"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264452912"}]},"ts":"1676264452912"} 2023-02-13 05:00:52,914 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264452912"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264452912"}]},"ts":"1676264452912"} 2023-02-13 05:00:52,914 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264452912"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264452912"}]},"ts":"1676264452912"} 2023-02-13 05:00:52,914 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264452912"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264452912"}]},"ts":"1676264452912"} 2023-02-13 05:00:52,914 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264452912"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264452912"}]},"ts":"1676264452912"} 2023-02-13 05:00:52,914 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264452912"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264452912"}]},"ts":"1676264452912"} 2023-02-13 05:00:52,915 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 05:00:52,916 INFO [PEWorker-4] hbase.MetaTableAccessor(1810): Overwritten 6 regions to Meta 2023-02-13 05:00:52,916 DEBUG [PEWorker-4] hbase.MetaTableAccessor(1811): Overwritten regions: [{ENCODED => 11ed57e84fcc1f7d47e9ad2ed09021f8, NAME => 'testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 347ea8719201fbf1ee1e731b16986087, NAME => 'testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => 7017e1f9205c2ebde9f23df4733114d5, NAME => 'testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.', STARTKEY => '5', ENDKEY => ''}, {ENCODED => d063b19c8c76cc04b197243b846f83ab, NAME => 'testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => d429d785f30801586e3db3ae3a9428b8, NAME => 'testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => ee2d32fb6918e03288e862be581d9bcb, NAME => 'testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.', STARTKEY => '4', ENDKEY => '5'}] 2023-02-13 05:00:52,916 INFO [PEWorker-4] procedure.RestoreSnapshotProcedure(447): Restore snapshot={ ss=snaptb1-1676264436786 table=testRestoreSnapshot-1676264436786 type=DISABLED ttl=0 } on table=testRestoreSnapshot-1676264436786 completed! 2023-02-13 05:00:52,918 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=147, state=SUCCESS; RestoreSnapshotProcedure (table=testRestoreSnapshot-1676264436786 snapshot=name: "snaptb1-1676264436786" table: "testRestoreSnapshot-1676264436786" creation_time: 1676264447622 type: DISABLED version: 2 ttl: 0 ) in 123 msec 2023-02-13 05:00:53,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=147 2023-02-13 05:00:53,049 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676264436786, procId: 147 completed 2023-02-13 05:00:53,049 INFO [Listener at localhost/42165] client.HBaseAdmin(2618): Deleting restore-failsafe snapshot: hbase-failsafe-snaptb1-1676264436786-1676264452516 2023-02-13 05:00:53,050 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "hbase-failsafe-snaptb1-1676264436786-1676264452516" 2023-02-13 05:00:53,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(333): Deleting snapshot: hbase-failsafe-snaptb1-1676264436786-1676264452516 2023-02-13 05:00:53,052 INFO [Listener at localhost/42165] client.HBaseAdmin$14(830): Started enable of testRestoreSnapshot-1676264436786 2023-02-13 05:00:53,053 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testRestoreSnapshot-1676264436786 2023-02-13 05:00:53,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=148, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:53,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-13 05:00:53,056 INFO [PEWorker-5] procedure.EnableTableProcedure(296): Attempting to enable the table testRestoreSnapshot-1676264436786 2023-02-13 05:00:53,056 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264453056"}]},"ts":"1676264453056"} 2023-02-13 05:00:53,057 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=ENABLING in hbase:meta 2023-02-13 05:00:53,061 DEBUG [PEWorker-5] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 05:00:53,061 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN}, {pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN}, {pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN}, {pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN}, {pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN}, {pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN}] 2023-02-13 05:00:53,062 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN 2023-02-13 05:00:53,062 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN 2023-02-13 05:00:53,063 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN 2023-02-13 05:00:53,063 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN 2023-02-13 05:00:53,063 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=149, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 05:00:53,063 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=151, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 05:00:53,063 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN 2023-02-13 05:00:53,063 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=150, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 05:00:53,063 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=152, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 05:00:53,064 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN 2023-02-13 05:00:53,064 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=154, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 05:00:53,064 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=153, ppid=148, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN; state=CLOSED, location=null; forceNewPlan=false, retain=false 2023-02-13 05:00:53,213 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 05:00:53,213 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 05:00:53,213 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 05:00:53,214 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 05:00:53,214 DEBUG [jenkins-hbase4:45905] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 05:00:53,217 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:53,217 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:53,217 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264453217"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264453217"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264453217"}]},"ts":"1676264453217"} 2023-02-13 05:00:53,217 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:53,217 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:53,217 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:53,217 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264453217"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264453217"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264453217"}]},"ts":"1676264453217"} 2023-02-13 05:00:53,217 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264453217"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264453217"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264453217"}]},"ts":"1676264453217"} 2023-02-13 05:00:53,217 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264453217"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264453217"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264453217"}]},"ts":"1676264453217"} 2023-02-13 05:00:53,217 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264453217"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264453217"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264453217"}]},"ts":"1676264453217"} 2023-02-13 05:00:53,219 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=155, ppid=152, state=RUNNABLE; OpenRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:53,219 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=156, ppid=154, state=RUNNABLE; OpenRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:53,220 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=157, ppid=153, state=RUNNABLE; OpenRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:53,220 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=158, ppid=151, state=RUNNABLE; OpenRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:53,221 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=159, ppid=150, state=RUNNABLE; OpenRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:53,222 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:53,222 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264453221"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264453221"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264453221"}]},"ts":"1676264453221"} 2023-02-13 05:00:53,223 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=160, ppid=149, state=RUNNABLE; OpenRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:53,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-13 05:00:53,373 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:53,373 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 347ea8719201fbf1ee1e731b16986087, NAME => 'testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 05:00:53,373 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:53,374 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:53,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:53,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ee2d32fb6918e03288e862be581d9bcb, NAME => 'testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 05:00:53,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:53,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:53,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:53,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:53,374 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:53,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:53,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:53,374 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7017e1f9205c2ebde9f23df4733114d5, NAME => 'testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 05:00:53,375 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:53,375 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:53,375 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:53,375 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:53,375 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:53,375 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:53,376 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:53,376 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf 2023-02-13 05:00:53,376 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf 2023-02-13 05:00:53,376 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf 2023-02-13 05:00:53,376 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf 2023-02-13 05:00:53,376 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 347ea8719201fbf1ee1e731b16986087 columnFamilyName cf 2023-02-13 05:00:53,377 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee2d32fb6918e03288e862be581d9bcb columnFamilyName cf 2023-02-13 05:00:53,377 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf 2023-02-13 05:00:53,377 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf 2023-02-13 05:00:53,377 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7017e1f9205c2ebde9f23df4733114d5 columnFamilyName cf 2023-02-13 05:00:53,395 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d15ef8effb63426f88cf28b863b418da 2023-02-13 05:00:53,395 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-3f25ec8511e84234a9c7a722ca070d0b 2023-02-13 05:00:53,396 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-519efdc2aee8432c80befe61da0da81c 2023-02-13 05:00:53,404 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d318209d74c44b3fb2d00363c5890672 2023-02-13 05:00:53,404 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(310): Store=7017e1f9205c2ebde9f23df4733114d5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:53,404 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-b0b32b9f22144056b97173b2a69002a5 2023-02-13 05:00:53,405 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(310): Store=347ea8719201fbf1ee1e731b16986087/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:53,405 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:53,405 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:53,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:53,407 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:53,409 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-d3866b21c6944cf0882c65fec182cb40 2023-02-13 05:00:53,409 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(310): Store=ee2d32fb6918e03288e862be581d9bcb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:53,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:53,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:53,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:53,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:53,411 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 347ea8719201fbf1ee1e731b16986087; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59096502, jitterRate=-0.1193934977054596}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:53,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 347ea8719201fbf1ee1e731b16986087: Running coprocessor pre-open hook at 1676264453374Writing region info on filesystem at 1676264453374Initializing all the Stores at 1676264453375 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264453375Cleaning up temporary data from old regions at 1676264453407 (+32 ms)Cleaning up detritus from prior splits at 1676264453408 (+1 ms)Running coprocessor post-open hooks at 1676264453411 (+3 ms)Region opened successfully at 1676264453411 2023-02-13 05:00:53,412 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7017e1f9205c2ebde9f23df4733114d5; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59487140, jitterRate=-0.11357253789901733}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:53,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7017e1f9205c2ebde9f23df4733114d5: Running coprocessor pre-open hook at 1676264453375Writing region info on filesystem at 1676264453375Initializing all the Stores at 1676264453375Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264453375Cleaning up temporary data from old regions at 1676264453407 (+32 ms)Cleaning up detritus from prior splits at 1676264453407Running coprocessor post-open hooks at 1676264453412 (+5 ms)Region opened successfully at 1676264453412 2023-02-13 05:00:53,413 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087., pid=155, masterSystemTime=1676264453370 2023-02-13 05:00:53,413 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5., pid=156, masterSystemTime=1676264453372 2023-02-13 05:00:53,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:53,415 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:53,415 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:53,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d429d785f30801586e3db3ae3a9428b8, NAME => 'testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 05:00:53,415 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:53,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:53,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:53,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:53,416 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=152 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:53,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:53,416 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:53,416 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:53,416 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264453416"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264453416"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264453416"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264453416"}]},"ts":"1676264453416"} 2023-02-13 05:00:53,416 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:53,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 11ed57e84fcc1f7d47e9ad2ed09021f8, NAME => 'testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 05:00:53,417 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=154 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:53,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:53,417 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264453417"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264453417"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264453417"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264453417"}]},"ts":"1676264453417"} 2023-02-13 05:00:53,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:53,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:53,417 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:53,418 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ee2d32fb6918e03288e862be581d9bcb; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66818332, jitterRate=-0.004329264163970947}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:53,418 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ee2d32fb6918e03288e862be581d9bcb: Running coprocessor pre-open hook at 1676264453374Writing region info on filesystem at 1676264453374Initializing all the Stores at 1676264453375 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264453375Cleaning up temporary data from old regions at 1676264453412 (+37 ms)Cleaning up detritus from prior splits at 1676264453412Running coprocessor post-open hooks at 1676264453418 (+6 ms)Region opened successfully at 1676264453418 2023-02-13 05:00:53,419 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb., pid=157, masterSystemTime=1676264453372 2023-02-13 05:00:53,419 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:53,420 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:53,421 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf 2023-02-13 05:00:53,421 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf 2023-02-13 05:00:53,422 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=153 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:53,421 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:53,421 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf 2023-02-13 05:00:53,422 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=155, resume processing ppid=152 2023-02-13 05:00:53,422 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=155, ppid=152, state=SUCCESS; OpenRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,34363,1676264434089 in 201 msec 2023-02-13 05:00:53,422 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=156, resume processing ppid=154 2023-02-13 05:00:53,422 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=156, ppid=154, state=SUCCESS; OpenRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,42589,1676264434015 in 200 msec 2023-02-13 05:00:53,422 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:53,423 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:53,422 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264453422"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264453422"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264453422"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264453422"}]},"ts":"1676264453422"} 2023-02-13 05:00:53,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d063b19c8c76cc04b197243b846f83ab, NAME => 'testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 05:00:53,422 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d429d785f30801586e3db3ae3a9428b8 columnFamilyName cf 2023-02-13 05:00:53,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:53,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:53,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:53,423 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:53,422 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf 2023-02-13 05:00:53,424 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 11ed57e84fcc1f7d47e9ad2ed09021f8 columnFamilyName cf 2023-02-13 05:00:53,425 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=152, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN in 361 msec 2023-02-13 05:00:53,425 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=154, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN in 361 msec 2023-02-13 05:00:53,427 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=157, resume processing ppid=153 2023-02-13 05:00:53,427 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=157, ppid=153, state=SUCCESS; OpenRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,39885,1676264434059 in 205 msec 2023-02-13 05:00:53,428 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=153, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN in 366 msec 2023-02-13 05:00:53,432 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:53,433 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf 2023-02-13 05:00:53,433 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf 2023-02-13 05:00:53,434 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d063b19c8c76cc04b197243b846f83ab columnFamilyName cf 2023-02-13 05:00:53,444 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-3ae4957471034eeb9ed26ec1a046807b 2023-02-13 05:00:53,447 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-811ac31223674808858dc07f9153be03 2023-02-13 05:00:53,455 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-075fd0e105614d9aaf0995ac7325a2fb 2023-02-13 05:00:53,460 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-6414549fb99340299c9cb4b42e13bd55 2023-02-13 05:00:53,460 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(310): Store=11ed57e84fcc1f7d47e9ad2ed09021f8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:53,461 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-b95f98eee2d241bea094905381835a03 2023-02-13 05:00:53,461 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(310): Store=d429d785f30801586e3db3ae3a9428b8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:53,461 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:53,462 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:53,463 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:53,463 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:53,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:53,466 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:53,467 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d429d785f30801586e3db3ae3a9428b8; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68140882, jitterRate=0.01537826657295227}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:53,467 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 11ed57e84fcc1f7d47e9ad2ed09021f8; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60442618, jitterRate=-0.09933480620384216}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:53,467 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d429d785f30801586e3db3ae3a9428b8: Running coprocessor pre-open hook at 1676264453416Writing region info on filesystem at 1676264453416Initializing all the Stores at 1676264453419 (+3 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264453419Cleaning up temporary data from old regions at 1676264453464 (+45 ms)Cleaning up detritus from prior splits at 1676264453464Running coprocessor post-open hooks at 1676264453467 (+3 ms)Region opened successfully at 1676264453467 2023-02-13 05:00:53,467 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 11ed57e84fcc1f7d47e9ad2ed09021f8: Running coprocessor pre-open hook at 1676264453417Writing region info on filesystem at 1676264453417Initializing all the Stores at 1676264453419 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264453419Cleaning up temporary data from old regions at 1676264453464 (+45 ms)Cleaning up detritus from prior splits at 1676264453464Running coprocessor post-open hooks at 1676264453467 (+3 ms)Region opened successfully at 1676264453467 2023-02-13 05:00:53,468 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8., pid=160, masterSystemTime=1676264453370 2023-02-13 05:00:53,468 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8., pid=158, masterSystemTime=1676264453372 2023-02-13 05:00:53,469 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:53,469 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:53,470 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=149 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:53,470 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264453470"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264453470"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264453470"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264453470"}]},"ts":"1676264453470"} 2023-02-13 05:00:53,470 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:53,470 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:53,470 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=151 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:53,470 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264453470"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264453470"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264453470"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264453470"}]},"ts":"1676264453470"} 2023-02-13 05:00:53,472 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-5ba29d67350f40888be6bfe3fec7f441 2023-02-13 05:00:53,472 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(310): Store=d063b19c8c76cc04b197243b846f83ab/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:53,473 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=160, resume processing ppid=149 2023-02-13 05:00:53,473 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=160, ppid=149, state=SUCCESS; OpenRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,34363,1676264434089 in 248 msec 2023-02-13 05:00:53,473 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:53,474 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=158, resume processing ppid=151 2023-02-13 05:00:53,474 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=158, ppid=151, state=SUCCESS; OpenRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,42589,1676264434015 in 252 msec 2023-02-13 05:00:53,474 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=149, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN in 412 msec 2023-02-13 05:00:53,475 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:53,475 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=151, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN in 413 msec 2023-02-13 05:00:53,477 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:53,478 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d063b19c8c76cc04b197243b846f83ab; next sequenceid=25; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68365008, jitterRate=0.01871800422668457}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:53,478 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d063b19c8c76cc04b197243b846f83ab: Running coprocessor pre-open hook at 1676264453423Writing region info on filesystem at 1676264453423Initializing all the Stores at 1676264453424 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264453424Cleaning up temporary data from old regions at 1676264453475 (+51 ms)Cleaning up detritus from prior splits at 1676264453476 (+1 ms)Running coprocessor post-open hooks at 1676264453478 (+2 ms)Region opened successfully at 1676264453478 2023-02-13 05:00:53,478 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab., pid=159, masterSystemTime=1676264453372 2023-02-13 05:00:53,480 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:53,480 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:53,481 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=150 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=OPEN, openSeqNum=25, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:53,481 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264453480"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264453480"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264453480"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264453480"}]},"ts":"1676264453480"} 2023-02-13 05:00:53,483 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=159, resume processing ppid=150 2023-02-13 05:00:53,484 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=159, ppid=150, state=SUCCESS; OpenRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,39885,1676264434059 in 261 msec 2023-02-13 05:00:53,486 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=150, resume processing ppid=148 2023-02-13 05:00:53,486 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=150, ppid=148, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN in 423 msec 2023-02-13 05:00:53,486 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264453486"}]},"ts":"1676264453486"} 2023-02-13 05:00:53,487 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=ENABLED in hbase:meta 2023-02-13 05:00:53,489 INFO [PEWorker-1] procedure.EnableTableProcedure(310): Table 'testRestoreSnapshot-1676264436786' was successfully enabled. 2023-02-13 05:00:53,491 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=148, state=SUCCESS; EnableTableProcedure table=testRestoreSnapshot-1676264436786 in 437 msec 2023-02-13 05:00:53,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=148 2023-02-13 05:00:53,808 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testRestoreSnapshot-1676264436786, procId: 148 completed 2023-02-13 05:00:53,808 DEBUG [Listener at localhost/42165] zookeeper.ReadOnlyZKClient(139): Connect 0x03516549 to 127.0.0.1:63943 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 05:00:53,813 DEBUG [Listener at localhost/42165] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c5e1909, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 05:00:53,815 DEBUG [hconnection-0x418a69a3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:53,818 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44476, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:53,821 DEBUG [hconnection-0x418a69a3-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:53,823 INFO [RS-EventLoopGroup-5-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33528, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:53,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,937 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b0f3f9f to 127.0.0.1:63943 2023-02-13 05:00:53,937 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:00:53,938 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x05da8053 to 127.0.0.1:63943 2023-02-13 05:00:53,938 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:00:53,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:53,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,098 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 05:00:54,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,271 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 05:00:54,274 DEBUG [hconnection-0x418a69a3-shared-pool-2] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:54,276 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59596, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:54,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,471 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 05:00:54,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,690 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 05:00:54,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,884 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 05:00:54,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:54,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:55,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:56,933 INFO [Listener at localhost/42165] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676264436786 2023-02-13 05:00:56,933 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676264436786 2023-02-13 05:00:56,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=161, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:56,937 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264456937"}]},"ts":"1676264456937"} 2023-02-13 05:00:56,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-13 05:00:56,938 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=DISABLING in hbase:meta 2023-02-13 05:00:56,940 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676264436786 to state=DISABLING 2023-02-13 05:00:56,941 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN}, {pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN}, {pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN}, {pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN}, {pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN}, {pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN}] 2023-02-13 05:00:56,943 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=163, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN 2023-02-13 05:00:56,943 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=164, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN 2023-02-13 05:00:56,943 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=165, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN 2023-02-13 05:00:56,943 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=162, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN 2023-02-13 05:00:56,943 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=167, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN 2023-02-13 05:00:56,943 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:56,943 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:56,944 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264456943"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264456943"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264456943"}]},"ts":"1676264456943"} 2023-02-13 05:00:56,944 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264456943"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264456943"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264456943"}]},"ts":"1676264456943"} 2023-02-13 05:00:56,944 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:56,944 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:56,944 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:56,944 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264456944"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264456944"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264456944"}]},"ts":"1676264456944"} 2023-02-13 05:00:56,944 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264456944"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264456944"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264456944"}]},"ts":"1676264456944"} 2023-02-13 05:00:56,944 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264456944"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264456944"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264456944"}]},"ts":"1676264456944"} 2023-02-13 05:00:56,945 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=168, ppid=163, state=RUNNABLE; CloseRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:56,945 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=169, ppid=164, state=RUNNABLE; CloseRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:56,946 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=170, ppid=167, state=RUNNABLE; CloseRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:56,947 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=171, ppid=162, state=RUNNABLE; CloseRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:56,947 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=166, ppid=161, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN 2023-02-13 05:00:56,947 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=172, ppid=165, state=RUNNABLE; CloseRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:56,948 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:56,948 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264456948"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264456948"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264456948"}]},"ts":"1676264456948"} 2023-02-13 05:00:56,950 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=173, ppid=166, state=RUNNABLE; CloseRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:57,097 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:57,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d063b19c8c76cc04b197243b846f83ab, disabling compactions & flushes 2023-02-13 05:00:57,098 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:57,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:57,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. after waiting 0 ms 2023-02-13 05:00:57,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:57,099 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:57,099 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:57,100 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d429d785f30801586e3db3ae3a9428b8, disabling compactions & flushes 2023-02-13 05:00:57,100 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7017e1f9205c2ebde9f23df4733114d5, disabling compactions & flushes 2023-02-13 05:00:57,100 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:57,102 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:57,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:57,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:57,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. after waiting 0 ms 2023-02-13 05:00:57,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:57,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. after waiting 0 ms 2023-02-13 05:00:57,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:57,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 05:00:57,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 05:00:57,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:57,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d063b19c8c76cc04b197243b846f83ab: Waiting for close lock at 1676264457097Running coprocessor pre-close hooks at 1676264457097Disabling compacts and flushes for region at 1676264457097Disabling writes for close at 1676264457098 (+1 ms)Writing region close event to WAL at 1676264457103 (+5 ms)Running coprocessor post-close hooks at 1676264457111 (+8 ms)Closed at 1676264457111 2023-02-13 05:00:57,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:57,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d429d785f30801586e3db3ae3a9428b8: Waiting for close lock at 1676264457099Running coprocessor pre-close hooks at 1676264457099Disabling compacts and flushes for region at 1676264457099Disabling writes for close at 1676264457103 (+4 ms)Writing region close event to WAL at 1676264457108 (+5 ms)Running coprocessor post-close hooks at 1676264457111 (+3 ms)Closed at 1676264457112 (+1 ms) 2023-02-13 05:00:57,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 05:00:57,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:57,113 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7017e1f9205c2ebde9f23df4733114d5: Waiting for close lock at 1676264457099Running coprocessor pre-close hooks at 1676264457099Disabling compacts and flushes for region at 1676264457099Disabling writes for close at 1676264457103 (+4 ms)Writing region close event to WAL at 1676264457108 (+5 ms)Running coprocessor post-close hooks at 1676264457112 (+4 ms)Closed at 1676264457112 2023-02-13 05:00:57,113 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:57,113 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:57,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ee2d32fb6918e03288e862be581d9bcb, disabling compactions & flushes 2023-02-13 05:00:57,114 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:57,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:57,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. after waiting 0 ms 2023-02-13 05:00:57,114 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:57,114 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=163 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=CLOSED 2023-02-13 05:00:57,114 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264457114"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264457114"}]},"ts":"1676264457114"} 2023-02-13 05:00:57,114 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:57,114 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:57,115 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 347ea8719201fbf1ee1e731b16986087, disabling compactions & flushes 2023-02-13 05:00:57,118 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=162 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=CLOSED 2023-02-13 05:00:57,118 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264457118"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264457118"}]},"ts":"1676264457118"} 2023-02-13 05:00:57,118 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:57,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:57,118 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:57,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. after waiting 0 ms 2023-02-13 05:00:57,118 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:57,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:57,119 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=167 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=CLOSED 2023-02-13 05:00:57,119 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 11ed57e84fcc1f7d47e9ad2ed09021f8, disabling compactions & flushes 2023-02-13 05:00:57,119 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:57,119 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264457119"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264457119"}]},"ts":"1676264457119"} 2023-02-13 05:00:57,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:57,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. after waiting 0 ms 2023-02-13 05:00:57,120 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:57,130 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=168, resume processing ppid=163 2023-02-13 05:00:57,130 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=168, ppid=163, state=SUCCESS; CloseRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,39885,1676264434059 in 174 msec 2023-02-13 05:00:57,130 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=171, resume processing ppid=162 2023-02-13 05:00:57,130 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=171, ppid=162, state=SUCCESS; CloseRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,34363,1676264434089 in 177 msec 2023-02-13 05:00:57,131 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=163, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN in 189 msec 2023-02-13 05:00:57,131 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=162, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN in 189 msec 2023-02-13 05:00:57,133 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 05:00:57,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 05:00:57,134 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:57,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 11ed57e84fcc1f7d47e9ad2ed09021f8: Waiting for close lock at 1676264457119Running coprocessor pre-close hooks at 1676264457119Disabling compacts and flushes for region at 1676264457119Disabling writes for close at 1676264457120 (+1 ms)Writing region close event to WAL at 1676264457124 (+4 ms)Running coprocessor post-close hooks at 1676264457134 (+10 ms)Closed at 1676264457134 2023-02-13 05:00:57,134 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/recovered.edits/27.seqid, newMaxSeqId=27, maxSeqId=24 2023-02-13 05:00:57,135 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:57,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 347ea8719201fbf1ee1e731b16986087: Waiting for close lock at 1676264457114Running coprocessor pre-close hooks at 1676264457114Disabling compacts and flushes for region at 1676264457114Disabling writes for close at 1676264457118 (+4 ms)Writing region close event to WAL at 1676264457129 (+11 ms)Running coprocessor post-close hooks at 1676264457135 (+6 ms)Closed at 1676264457135 2023-02-13 05:00:57,135 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:57,135 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ee2d32fb6918e03288e862be581d9bcb: Waiting for close lock at 1676264457113Running coprocessor pre-close hooks at 1676264457113Disabling compacts and flushes for region at 1676264457113Disabling writes for close at 1676264457114 (+1 ms)Writing region close event to WAL at 1676264457118 (+4 ms)Running coprocessor post-close hooks at 1676264457135 (+17 ms)Closed at 1676264457135 2023-02-13 05:00:57,140 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:57,140 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=170, resume processing ppid=167 2023-02-13 05:00:57,141 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=164 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=CLOSED 2023-02-13 05:00:57,141 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=170, ppid=167, state=SUCCESS; CloseRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,42589,1676264434015 in 188 msec 2023-02-13 05:00:57,141 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264457141"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264457141"}]},"ts":"1676264457141"} 2023-02-13 05:00:57,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:57,141 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=166 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=CLOSED 2023-02-13 05:00:57,141 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264457141"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264457141"}]},"ts":"1676264457141"} 2023-02-13 05:00:57,142 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:57,142 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=167, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN in 200 msec 2023-02-13 05:00:57,142 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=165 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=CLOSED 2023-02-13 05:00:57,142 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264457142"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264457142"}]},"ts":"1676264457142"} 2023-02-13 05:00:57,144 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=169, resume processing ppid=164 2023-02-13 05:00:57,144 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=169, ppid=164, state=SUCCESS; CloseRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,42589,1676264434015 in 197 msec 2023-02-13 05:00:57,145 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=173, resume processing ppid=166 2023-02-13 05:00:57,145 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=173, ppid=166, state=SUCCESS; CloseRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,39885,1676264434059 in 193 msec 2023-02-13 05:00:57,146 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=172, resume processing ppid=165 2023-02-13 05:00:57,146 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=172, ppid=165, state=SUCCESS; CloseRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,34363,1676264434089 in 197 msec 2023-02-13 05:00:57,146 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=164, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN in 204 msec 2023-02-13 05:00:57,147 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=166, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN in 204 msec 2023-02-13 05:00:57,148 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=165, resume processing ppid=161 2023-02-13 05:00:57,148 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=165, ppid=161, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN in 205 msec 2023-02-13 05:00:57,149 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264457148"}]},"ts":"1676264457148"} 2023-02-13 05:00:57,150 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=DISABLED in hbase:meta 2023-02-13 05:00:57,152 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676264436786 to state=DISABLED 2023-02-13 05:00:57,154 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=161, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676264436786 in 220 msec 2023-02-13 05:00:57,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=161 2023-02-13 05:00:57,188 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676264436786, procId: 161 completed 2023-02-13 05:00:57,193 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,200 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=174, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,202 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=174, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-13 05:00:57,209 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:57,209 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:57,209 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:57,209 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:57,209 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:57,209 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:57,212 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf, FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/recovered.edits] 2023-02-13 05:00:57,212 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf, FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/recovered.edits] 2023-02-13 05:00:57,212 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf, FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/recovered.edits] 2023-02-13 05:00:57,213 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf, FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/recovered.edits] 2023-02-13 05:00:57,212 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf, FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/recovered.edits] 2023-02-13 05:00:57,213 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf, FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/recovered.edits] 2023-02-13 05:00:57,217 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-811ac31223674808858dc07f9153be03 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-811ac31223674808858dc07f9153be03 2023-02-13 05:00:57,217 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-519efdc2aee8432c80befe61da0da81c to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-519efdc2aee8432c80befe61da0da81c 2023-02-13 05:00:57,217 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-3ae4957471034eeb9ed26ec1a046807b to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-3ae4957471034eeb9ed26ec1a046807b 2023-02-13 05:00:57,217 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-075fd0e105614d9aaf0995ac7325a2fb to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-075fd0e105614d9aaf0995ac7325a2fb 2023-02-13 05:00:57,217 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-3f25ec8511e84234a9c7a722ca070d0b to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-3f25ec8511e84234a9c7a722ca070d0b 2023-02-13 05:00:57,217 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d15ef8effb63426f88cf28b863b418da to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d15ef8effb63426f88cf28b863b418da 2023-02-13 05:00:57,218 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-b0b32b9f22144056b97173b2a69002a5 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-b0b32b9f22144056b97173b2a69002a5 2023-02-13 05:00:57,219 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-6414549fb99340299c9cb4b42e13bd55 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-6414549fb99340299c9cb4b42e13bd55 2023-02-13 05:00:57,219 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-b95f98eee2d241bea094905381835a03 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-b95f98eee2d241bea094905381835a03 2023-02-13 05:00:57,219 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-5ba29d67350f40888be6bfe3fec7f441 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-5ba29d67350f40888be6bfe3fec7f441 2023-02-13 05:00:57,219 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d318209d74c44b3fb2d00363c5890672 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d318209d74c44b3fb2d00363c5890672 2023-02-13 05:00:57,220 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-d3866b21c6944cf0882c65fec182cb40 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-d3866b21c6944cf0882c65fec182cb40 2023-02-13 05:00:57,223 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/recovered.edits/27.seqid to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/recovered.edits/27.seqid 2023-02-13 05:00:57,223 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/recovered.edits/27.seqid to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/recovered.edits/27.seqid 2023-02-13 05:00:57,223 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/recovered.edits/27.seqid to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/recovered.edits/27.seqid 2023-02-13 05:00:57,223 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:57,224 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/recovered.edits/27.seqid to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/recovered.edits/27.seqid 2023-02-13 05:00:57,224 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:57,224 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:57,224 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:57,224 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/recovered.edits/27.seqid to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/recovered.edits/27.seqid 2023-02-13 05:00:57,225 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/recovered.edits/27.seqid to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/recovered.edits/27.seqid 2023-02-13 05:00:57,225 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:57,225 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:57,225 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676264436786 regions 2023-02-13 05:00:57,225 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6 2023-02-13 05:00:57,226 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf] 2023-02-13 05:00:57,228 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 2023-02-13 05:00:57,229 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 2023-02-13 05:00:57,230 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 2023-02-13 05:00:57,231 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 2023-02-13 05:00:57,232 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c 2023-02-13 05:00:57,233 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 2023-02-13 05:00:57,234 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 2023-02-13 05:00:57,235 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 2023-02-13 05:00:57,236 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 2023-02-13 05:00:57,237 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b 2023-02-13 05:00:57,238 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 2023-02-13 05:00:57,239 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 2023-02-13 05:00:57,240 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6 2023-02-13 05:00:57,242 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=174, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,263 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676264436786 from hbase:meta 2023-02-13 05:00:57,265 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676264436786' descriptor. 2023-02-13 05:00:57,266 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=174, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,266 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676264436786' from region states. 2023-02-13 05:00:57,266 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264457266"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:57,266 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264457266"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:57,266 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264457266"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:57,266 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264457266"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:57,266 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264457266"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:57,267 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264457266"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:57,268 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 05:00:57,268 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d429d785f30801586e3db3ae3a9428b8, NAME => 'testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => d063b19c8c76cc04b197243b846f83ab, NAME => 'testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 11ed57e84fcc1f7d47e9ad2ed09021f8, NAME => 'testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 347ea8719201fbf1ee1e731b16986087, NAME => 'testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => ee2d32fb6918e03288e862be581d9bcb, NAME => 'testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 7017e1f9205c2ebde9f23df4733114d5, NAME => 'testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.', STARTKEY => '5', ENDKEY => ''}] 2023-02-13 05:00:57,268 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676264436786' as deleted. 2023-02-13 05:00:57,268 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676264457268"}]},"ts":"9223372036854775807"} 2023-02-13 05:00:57,270 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676264436786 state from META 2023-02-13 05:00:57,271 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=174, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,272 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=174, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676264436786 in 77 msec 2023-02-13 05:00:57,287 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 05:00:57,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=174 2023-02-13 05:00:57,454 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676264436786, procId: 174 completed 2023-02-13 05:00:57,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=175, state=RUNNABLE:CLONE_SNAPSHOT_PRE_OPERATION; CloneSnapshotProcedure (table=testRestoreSnapshot-1676264436786 snapshot=name: "snaptb1-1676264436786" table: "testRestoreSnapshot-1676264436786" creation_time: 1676264447622 type: DISABLED version: 2 ttl: 0 ) 2023-02-13 05:00:57,481 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(750): Clone snapshot=snaptb1-1676264436786 as table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-13 05:00:57,496 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/.tabledesc/.tableinfo.0000000001 2023-02-13 05:00:57,498 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(177): starting restore table regions using snapshot=name: "snaptb1-1676264436786" table: "testRestoreSnapshot-1676264436786" creation_time: 1676264447622 type: DISABLED version: 2 ttl: 0 2023-02-13 05:00:57,498 DEBUG [PEWorker-3] snapshot.RestoreSnapshotHelper(785): get table regions: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,499 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,501 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,503 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,505 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,507 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,509 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,511 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,513 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,515 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,517 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,519 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,521 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,523 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:57,523 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:57,523 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:57,523 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:57,523 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:57,523 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(239): region to add: d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:57,523 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=d063b19c8c76cc04b197243b846f83ab as d063b19c8c76cc04b197243b846f83ab in snapshot snaptb1-1676264436786 2023-02-13 05:00:57,523 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=347ea8719201fbf1ee1e731b16986087 as 347ea8719201fbf1ee1e731b16986087 in snapshot snaptb1-1676264436786 2023-02-13 05:00:57,523 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=11ed57e84fcc1f7d47e9ad2ed09021f8 as 11ed57e84fcc1f7d47e9ad2ed09021f8 in snapshot snaptb1-1676264436786 2023-02-13 05:00:57,523 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=7017e1f9205c2ebde9f23df4733114d5 as 7017e1f9205c2ebde9f23df4733114d5 in snapshot snaptb1-1676264436786 2023-02-13 05:00:57,523 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=ee2d32fb6918e03288e862be581d9bcb as ee2d32fb6918e03288e862be581d9bcb in snapshot snaptb1-1676264436786 2023-02-13 05:00:57,523 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(585): clone region=d429d785f30801586e3db3ae3a9428b8 as d429d785f30801586e3db3ae3a9428b8 in snapshot snaptb1-1676264436786 2023-02-13 05:00:57,525 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(7671): creating {ENCODED => 7017e1f9205c2ebde9f23df4733114d5, NAME => 'testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testRestoreSnapshot-1676264436786', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp 2023-02-13 05:00:57,525 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(7671): creating {ENCODED => 11ed57e84fcc1f7d47e9ad2ed09021f8, NAME => 'testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testRestoreSnapshot-1676264436786', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp 2023-02-13 05:00:57,526 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(7671): creating {ENCODED => d063b19c8c76cc04b197243b846f83ab, NAME => 'testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testRestoreSnapshot-1676264436786', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp 2023-02-13 05:00:57,526 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(7671): creating {ENCODED => d429d785f30801586e3db3ae3a9428b8, NAME => 'testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testRestoreSnapshot-1676264436786', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp 2023-02-13 05:00:57,526 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(7671): creating {ENCODED => ee2d32fb6918e03288e862be581d9bcb, NAME => 'testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testRestoreSnapshot-1676264436786', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp 2023-02-13 05:00:57,525 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(7671): creating {ENCODED => 347ea8719201fbf1ee1e731b16986087, NAME => 'testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testRestoreSnapshot-1676264436786', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp 2023-02-13 05:00:57,564 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:57,564 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:57,565 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 3f25ec8511e84234a9c7a722ca070d0b from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,565 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d15ef8effb63426f88cf28b863b418da from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,566 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:57,566 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 3ae4957471034eeb9ed26ec1a046807b from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,566 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:57,566 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 075fd0e105614d9aaf0995ac7325a2fb from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,567 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:57,567 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 519efdc2aee8432c80befe61da0da81c from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,568 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:57,568 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 811ac31223674808858dc07f9153be03 from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,570 INFO [RestoreSnapshot-pool-4] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d3866b21c6944cf0882c65fec182cb40 from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,571 INFO [RestoreSnapshot-pool-2] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 6414549fb99340299c9cb4b42e13bd55 from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,571 INFO [RestoreSnapshot-pool-0] snapshot.RestoreSnapshotHelper(628): Adding HFileLink 5ba29d67350f40888be6bfe3fec7f441 from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,571 INFO [RestoreSnapshot-pool-3] snapshot.RestoreSnapshotHelper(628): Adding HFileLink d318209d74c44b3fb2d00363c5890672 from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,572 INFO [RestoreSnapshot-pool-5] snapshot.RestoreSnapshotHelper(628): Adding HFileLink b95f98eee2d241bea094905381835a03 from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,572 INFO [RestoreSnapshot-pool-1] snapshot.RestoreSnapshotHelper(628): Adding HFileLink b0b32b9f22144056b97173b2a69002a5 from cloned region in snapshot snaptb1-1676264436786 to table=testRestoreSnapshot-1676264436786 2023-02-13 05:00:57,574 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1603): Closing 11ed57e84fcc1f7d47e9ad2ed09021f8, disabling compactions & flushes 2023-02-13 05:00:57,574 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:57,574 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:57,574 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. after waiting 0 ms 2023-02-13 05:00:57,574 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:57,574 INFO [RestoreSnapshot-pool-2] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:57,574 DEBUG [RestoreSnapshot-pool-2] regionserver.HRegion(1557): Region close journal for 11ed57e84fcc1f7d47e9ad2ed09021f8: Waiting for close lock at 1676264457574Disabling compacts and flushes for region at 1676264457574Disabling writes for close at 1676264457574Writing region close event to WAL at 1676264457574Closed at 1676264457574 2023-02-13 05:00:57,575 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1603): Closing ee2d32fb6918e03288e862be581d9bcb, disabling compactions & flushes 2023-02-13 05:00:57,575 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:57,575 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:57,575 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. after waiting 0 ms 2023-02-13 05:00:57,575 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:57,575 INFO [RestoreSnapshot-pool-4] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:57,575 DEBUG [RestoreSnapshot-pool-4] regionserver.HRegion(1557): Region close journal for ee2d32fb6918e03288e862be581d9bcb: Waiting for close lock at 1676264457574Disabling compacts and flushes for region at 1676264457574Disabling writes for close at 1676264457575 (+1 ms)Writing region close event to WAL at 1676264457575Closed at 1676264457575 2023-02-13 05:00:57,575 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1603): Closing d063b19c8c76cc04b197243b846f83ab, disabling compactions & flushes 2023-02-13 05:00:57,575 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:57,575 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:57,575 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. after waiting 0 ms 2023-02-13 05:00:57,575 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:57,575 INFO [RestoreSnapshot-pool-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:57,575 DEBUG [RestoreSnapshot-pool-0] regionserver.HRegion(1557): Region close journal for d063b19c8c76cc04b197243b846f83ab: Waiting for close lock at 1676264457575Disabling compacts and flushes for region at 1676264457575Disabling writes for close at 1676264457575Writing region close event to WAL at 1676264457575Closed at 1676264457575 2023-02-13 05:00:57,576 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1603): Closing d429d785f30801586e3db3ae3a9428b8, disabling compactions & flushes 2023-02-13 05:00:57,576 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:57,576 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:57,576 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. after waiting 0 ms 2023-02-13 05:00:57,576 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1603): Closing 347ea8719201fbf1ee1e731b16986087, disabling compactions & flushes 2023-02-13 05:00:57,576 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:57,576 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1603): Closing 7017e1f9205c2ebde9f23df4733114d5, disabling compactions & flushes 2023-02-13 05:00:57,576 INFO [RestoreSnapshot-pool-5] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:57,576 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:57,576 DEBUG [RestoreSnapshot-pool-5] regionserver.HRegion(1557): Region close journal for d429d785f30801586e3db3ae3a9428b8: Waiting for close lock at 1676264457576Disabling compacts and flushes for region at 1676264457576Disabling writes for close at 1676264457576Writing region close event to WAL at 1676264457576Closed at 1676264457576 2023-02-13 05:00:57,576 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:57,576 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:57,576 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:57,576 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. after waiting 0 ms 2023-02-13 05:00:57,576 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:57,576 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. after waiting 0 ms 2023-02-13 05:00:57,576 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:57,576 INFO [RestoreSnapshot-pool-1] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:57,576 INFO [RestoreSnapshot-pool-3] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:57,576 DEBUG [RestoreSnapshot-pool-1] regionserver.HRegion(1557): Region close journal for 347ea8719201fbf1ee1e731b16986087: Waiting for close lock at 1676264457576Disabling compacts and flushes for region at 1676264457576Disabling writes for close at 1676264457576Writing region close event to WAL at 1676264457576Closed at 1676264457576 2023-02-13 05:00:57,576 DEBUG [RestoreSnapshot-pool-3] regionserver.HRegion(1557): Region close journal for 7017e1f9205c2ebde9f23df4733114d5: Waiting for close lock at 1676264457576Disabling compacts and flushes for region at 1676264457576Disabling writes for close at 1676264457576Writing region close event to WAL at 1676264457576Closed at 1676264457576 2023-02-13 05:00:57,577 INFO [PEWorker-3] snapshot.RestoreSnapshotHelper(266): finishing restore table regions using snapshot=name: "snaptb1-1676264436786" table: "testRestoreSnapshot-1676264436786" creation_time: 1676264447622 type: DISABLED version: 2 ttl: 0 2023-02-13 05:00:57,577 INFO [PEWorker-3] procedure.CloneSnapshotProcedure$1(411): Clone snapshot=snaptb1-1676264436786 on table=testRestoreSnapshot-1676264436786 completed! 2023-02-13 05:00:57,581 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264457581"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264457581"}]},"ts":"1676264457581"} 2023-02-13 05:00:57,581 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264457581"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264457581"}]},"ts":"1676264457581"} 2023-02-13 05:00:57,581 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264457581"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264457581"}]},"ts":"1676264457581"} 2023-02-13 05:00:57,581 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264457581"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264457581"}]},"ts":"1676264457581"} 2023-02-13 05:00:57,581 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264457581"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264457581"}]},"ts":"1676264457581"} 2023-02-13 05:00:57,581 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264457581"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264457581"}]},"ts":"1676264457581"} 2023-02-13 05:00:57,583 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 05:00:57,584 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264457584"}]},"ts":"1676264457584"} 2023-02-13 05:00:57,585 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=ENABLING in hbase:meta 2023-02-13 05:00:57,592 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 05:00:57,592 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 05:00:57,593 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 05:00:57,593 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 05:00:57,593 DEBUG [PEWorker-3] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 05:00:57,593 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN}, {pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN}, {pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN}, {pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN}, {pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN}, {pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN}] 2023-02-13 05:00:57,595 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN 2023-02-13 05:00:57,595 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN 2023-02-13 05:00:57,595 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN 2023-02-13 05:00:57,595 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN 2023-02-13 05:00:57,595 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN 2023-02-13 05:00:57,596 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=181, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42589,1676264434015; forceNewPlan=false, retain=false 2023-02-13 05:00:57,596 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=178, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39885,1676264434059; forceNewPlan=false, retain=false 2023-02-13 05:00:57,596 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=180, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34363,1676264434089; forceNewPlan=false, retain=false 2023-02-13 05:00:57,596 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=179, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34363,1676264434089; forceNewPlan=false, retain=false 2023-02-13 05:00:57,596 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=177, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42589,1676264434015; forceNewPlan=false, retain=false 2023-02-13 05:00:57,597 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN 2023-02-13 05:00:57,597 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=176, ppid=175, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39885,1676264434059; forceNewPlan=false, retain=false 2023-02-13 05:00:57,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-13 05:00:57,746 INFO [jenkins-hbase4:45905] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 05:00:57,749 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:57,749 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:57,749 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264457749"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264457749"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264457749"}]},"ts":"1676264457749"} 2023-02-13 05:00:57,749 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:57,749 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:57,749 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:57,749 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264457749"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264457749"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264457749"}]},"ts":"1676264457749"} 2023-02-13 05:00:57,750 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264457749"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264457749"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264457749"}]},"ts":"1676264457749"} 2023-02-13 05:00:57,749 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264457749"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264457749"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264457749"}]},"ts":"1676264457749"} 2023-02-13 05:00:57,750 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264457749"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264457749"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264457749"}]},"ts":"1676264457749"} 2023-02-13 05:00:57,751 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=182, ppid=181, state=RUNNABLE; OpenRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:57,753 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=183, ppid=177, state=RUNNABLE; OpenRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:00:57,754 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=184, ppid=178, state=RUNNABLE; OpenRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:57,754 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=185, ppid=176, state=RUNNABLE; OpenRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:00:57,755 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=186, ppid=180, state=RUNNABLE; OpenRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:57,755 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:57,755 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264457755"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264457755"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264457755"}]},"ts":"1676264457755"} 2023-02-13 05:00:57,757 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=187, ppid=179, state=RUNNABLE; OpenRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:00:57,907 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:57,907 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:57,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 11ed57e84fcc1f7d47e9ad2ed09021f8, NAME => 'testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 05:00:57,907 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 7017e1f9205c2ebde9f23df4733114d5, NAME => 'testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 05:00:57,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:57,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:57,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:57,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:57,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:57,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:57,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:57,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:57,908 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:57,908 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 347ea8719201fbf1ee1e731b16986087, NAME => 'testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 05:00:57,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:57,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:57,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:57,909 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:57,909 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:57,909 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:57,910 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:57,910 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf 2023-02-13 05:00:57,910 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf 2023-02-13 05:00:57,910 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf 2023-02-13 05:00:57,910 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf 2023-02-13 05:00:57,910 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf 2023-02-13 05:00:57,910 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf 2023-02-13 05:00:57,910 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 7017e1f9205c2ebde9f23df4733114d5 columnFamilyName cf 2023-02-13 05:00:57,910 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 11ed57e84fcc1f7d47e9ad2ed09021f8 columnFamilyName cf 2023-02-13 05:00:57,911 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 347ea8719201fbf1ee1e731b16986087 columnFamilyName cf 2023-02-13 05:00:57,923 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d15ef8effb63426f88cf28b863b418da 2023-02-13 05:00:57,923 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-3ae4957471034eeb9ed26ec1a046807b 2023-02-13 05:00:57,923 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-519efdc2aee8432c80befe61da0da81c 2023-02-13 05:00:57,929 DEBUG [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-6414549fb99340299c9cb4b42e13bd55 2023-02-13 05:00:57,929 INFO [StoreOpener-11ed57e84fcc1f7d47e9ad2ed09021f8-1] regionserver.HStore(310): Store=11ed57e84fcc1f7d47e9ad2ed09021f8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:57,930 DEBUG [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d318209d74c44b3fb2d00363c5890672 2023-02-13 05:00:57,930 INFO [StoreOpener-7017e1f9205c2ebde9f23df4733114d5-1] regionserver.HStore(310): Store=7017e1f9205c2ebde9f23df4733114d5/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:57,930 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:57,930 DEBUG [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-b0b32b9f22144056b97173b2a69002a5 2023-02-13 05:00:57,930 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:57,930 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:57,931 INFO [StoreOpener-347ea8719201fbf1ee1e731b16986087-1] regionserver.HStore(310): Store=347ea8719201fbf1ee1e731b16986087/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:57,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:57,931 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:57,932 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:57,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:00:57,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:00:57,933 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:00:57,934 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 05:00:57,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 11ed57e84fcc1f7d47e9ad2ed09021f8; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62458533, jitterRate=-0.06929533183574677}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:57,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 05:00:57,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 11ed57e84fcc1f7d47e9ad2ed09021f8: Running coprocessor pre-open hook at 1676264457908Writing region info on filesystem at 1676264457908Initializing all the Stores at 1676264457909 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264457909Cleaning up temporary data from old regions at 1676264457931 (+22 ms)Cleaning up detritus from prior splits at 1676264457931Running coprocessor post-open hooks at 1676264457935 (+4 ms)Region opened successfully at 1676264457935 2023-02-13 05:00:57,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 05:00:57,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 7017e1f9205c2ebde9f23df4733114d5; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62876669, jitterRate=-0.06306461989879608}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:57,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 7017e1f9205c2ebde9f23df4733114d5: Running coprocessor pre-open hook at 1676264457908Writing region info on filesystem at 1676264457908Initializing all the Stores at 1676264457909 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264457909Cleaning up temporary data from old regions at 1676264457931 (+22 ms)Cleaning up detritus from prior splits at 1676264457931Running coprocessor post-open hooks at 1676264457935 (+4 ms)Region opened successfully at 1676264457935 2023-02-13 05:00:57,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 347ea8719201fbf1ee1e731b16986087; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75270421, jitterRate=0.1216166764497757}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:57,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 347ea8719201fbf1ee1e731b16986087: Running coprocessor pre-open hook at 1676264457909Writing region info on filesystem at 1676264457909Initializing all the Stores at 1676264457909Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264457909Cleaning up temporary data from old regions at 1676264457932 (+23 ms)Cleaning up detritus from prior splits at 1676264457932Running coprocessor post-open hooks at 1676264457935 (+3 ms)Region opened successfully at 1676264457935 2023-02-13 05:00:57,936 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8., pid=184, masterSystemTime=1676264457905 2023-02-13 05:00:57,936 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5., pid=182, masterSystemTime=1676264457904 2023-02-13 05:00:57,936 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087., pid=187, masterSystemTime=1676264457906 2023-02-13 05:00:57,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:57,937 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:00:57,937 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:57,937 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d429d785f30801586e3db3ae3a9428b8, NAME => 'testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 05:00:57,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:57,938 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=178 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:57,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:57,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:57,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:57,938 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:00:57,938 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:57,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => d063b19c8c76cc04b197243b846f83ab, NAME => 'testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 05:00:57,938 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=181 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:57,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:57,938 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264457938"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264457938"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264457938"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264457938"}]},"ts":"1676264457938"} 2023-02-13 05:00:57,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:57,938 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264457937"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264457937"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264457937"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264457937"}]},"ts":"1676264457937"} 2023-02-13 05:00:57,938 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:00:57,939 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:57,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:57,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ee2d32fb6918e03288e862be581d9bcb, NAME => 'testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 05:00:57,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:57,938 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:57,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:57,939 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=179 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:57,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testRestoreSnapshot-1676264436786 ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:57,939 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264457939"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264457939"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264457939"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264457939"}]},"ts":"1676264457939"} 2023-02-13 05:00:57,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:00:57,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:57,939 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:57,940 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:57,941 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:57,941 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf 2023-02-13 05:00:57,941 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf 2023-02-13 05:00:57,942 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d429d785f30801586e3db3ae3a9428b8 columnFamilyName cf 2023-02-13 05:00:57,942 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:57,942 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=182, resume processing ppid=181 2023-02-13 05:00:57,942 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=182, ppid=181, state=SUCCESS; OpenRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,42589,1676264434015 in 189 msec 2023-02-13 05:00:57,942 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf 2023-02-13 05:00:57,942 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf 2023-02-13 05:00:57,943 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d063b19c8c76cc04b197243b846f83ab columnFamilyName cf 2023-02-13 05:00:57,943 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=184, resume processing ppid=178 2023-02-13 05:00:57,943 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=184, ppid=178, state=SUCCESS; OpenRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,39885,1676264434059 in 186 msec 2023-02-13 05:00:57,943 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf 2023-02-13 05:00:57,943 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf 2023-02-13 05:00:57,943 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee2d32fb6918e03288e862be581d9bcb columnFamilyName cf 2023-02-13 05:00:57,946 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=181, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, ASSIGN in 349 msec 2023-02-13 05:00:57,946 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=187, resume processing ppid=179 2023-02-13 05:00:57,946 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=187, ppid=179, state=SUCCESS; OpenRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,34363,1676264434089 in 184 msec 2023-02-13 05:00:57,947 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=178, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, ASSIGN in 350 msec 2023-02-13 05:00:57,949 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=179, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, ASSIGN in 353 msec 2023-02-13 05:00:57,952 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-811ac31223674808858dc07f9153be03 2023-02-13 05:00:57,956 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-075fd0e105614d9aaf0995ac7325a2fb 2023-02-13 05:00:57,957 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-3f25ec8511e84234a9c7a722ca070d0b 2023-02-13 05:00:57,958 DEBUG [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-b95f98eee2d241bea094905381835a03 2023-02-13 05:00:57,958 INFO [StoreOpener-d429d785f30801586e3db3ae3a9428b8-1] regionserver.HStore(310): Store=d429d785f30801586e3db3ae3a9428b8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:57,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:57,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:57,961 DEBUG [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-5ba29d67350f40888be6bfe3fec7f441 2023-02-13 05:00:57,961 INFO [StoreOpener-d063b19c8c76cc04b197243b846f83ab-1] regionserver.HStore(310): Store=d063b19c8c76cc04b197243b846f83ab/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:57,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:00:57,961 DEBUG [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-d3866b21c6944cf0882c65fec182cb40 2023-02-13 05:00:57,961 INFO [StoreOpener-ee2d32fb6918e03288e862be581d9bcb-1] regionserver.HStore(310): Store=ee2d32fb6918e03288e862be581d9bcb/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:00:57,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:57,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:57,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:57,962 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:57,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:00:57,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:00:57,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 05:00:57,965 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d429d785f30801586e3db3ae3a9428b8; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60809800, jitterRate=-0.09386336803436279}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:57,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d429d785f30801586e3db3ae3a9428b8: Running coprocessor pre-open hook at 1676264457939Writing region info on filesystem at 1676264457939Initializing all the Stores at 1676264457940 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264457940Cleaning up temporary data from old regions at 1676264457959 (+19 ms)Cleaning up detritus from prior splits at 1676264457960 (+1 ms)Running coprocessor post-open hooks at 1676264457965 (+5 ms)Region opened successfully at 1676264457965 2023-02-13 05:00:57,965 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8., pid=185, masterSystemTime=1676264457905 2023-02-13 05:00:57,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 05:00:57,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/recovered.edits/15.seqid, newMaxSeqId=15, maxSeqId=-1 2023-02-13 05:00:57,966 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened d063b19c8c76cc04b197243b846f83ab; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66614705, jitterRate=-0.007363542914390564}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:57,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for d063b19c8c76cc04b197243b846f83ab: Running coprocessor pre-open hook at 1676264457939Writing region info on filesystem at 1676264457939Initializing all the Stores at 1676264457940 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264457940Cleaning up temporary data from old regions at 1676264457962 (+22 ms)Cleaning up detritus from prior splits at 1676264457963 (+1 ms)Running coprocessor post-open hooks at 1676264457966 (+3 ms)Region opened successfully at 1676264457966 2023-02-13 05:00:57,966 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ee2d32fb6918e03288e862be581d9bcb; next sequenceid=16; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63527333, jitterRate=-0.05336897075176239}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:00:57,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ee2d32fb6918e03288e862be581d9bcb: Running coprocessor pre-open hook at 1676264457939Writing region info on filesystem at 1676264457939Initializing all the Stores at 1676264457940 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264457940Cleaning up temporary data from old regions at 1676264457963 (+23 ms)Cleaning up detritus from prior splits at 1676264457963Running coprocessor post-open hooks at 1676264457967 (+4 ms)Region opened successfully at 1676264457967 2023-02-13 05:00:57,967 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:57,967 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:00:57,967 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab., pid=183, masterSystemTime=1676264457904 2023-02-13 05:00:57,967 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb., pid=186, masterSystemTime=1676264457906 2023-02-13 05:00:57,967 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=176 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:00:57,967 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264457967"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264457967"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264457967"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264457967"}]},"ts":"1676264457967"} 2023-02-13 05:00:57,968 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:57,968 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:00:57,968 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=177 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:00:57,968 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264457968"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264457968"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264457968"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264457968"}]},"ts":"1676264457968"} 2023-02-13 05:00:57,970 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=180 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=OPEN, openSeqNum=16, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:00:57,970 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264457970"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264457970"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264457970"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264457970"}]},"ts":"1676264457970"} 2023-02-13 05:00:57,970 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=185, resume processing ppid=176 2023-02-13 05:00:57,971 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=185, ppid=176, state=SUCCESS; OpenRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059 in 215 msec 2023-02-13 05:00:57,972 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=176, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, ASSIGN in 378 msec 2023-02-13 05:00:57,973 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=183, resume processing ppid=177 2023-02-13 05:00:57,973 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=183, ppid=177, state=SUCCESS; OpenRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,42589,1676264434015 in 220 msec 2023-02-13 05:00:57,974 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=186, resume processing ppid=180 2023-02-13 05:00:57,974 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=186, ppid=180, state=SUCCESS; OpenRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,34363,1676264434089 in 216 msec 2023-02-13 05:00:57,974 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=177, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, ASSIGN in 380 msec 2023-02-13 05:00:57,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:57,975 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:00:57,975 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=180, resume processing ppid=175 2023-02-13 05:00:57,975 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=180, ppid=175, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, ASSIGN in 381 msec 2023-02-13 05:00:57,976 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264457976"}]},"ts":"1676264457976"} 2023-02-13 05:00:57,977 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=ENABLED in hbase:meta 2023-02-13 05:00:57,981 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=175, state=SUCCESS; CloneSnapshotProcedure (table=testRestoreSnapshot-1676264436786 snapshot=name: "snaptb1-1676264436786" table: "testRestoreSnapshot-1676264436786" creation_time: 1676264447622 type: DISABLED version: 2 ttl: 0 ) in 503 msec 2023-02-13 05:00:58,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=175 2023-02-13 05:00:58,236 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: MODIFY, Table Name: default:testRestoreSnapshot-1676264436786, procId: 175 completed 2023-02-13 05:00:58,237 DEBUG [Listener at localhost/42165] zookeeper.ReadOnlyZKClient(139): Connect 0x6eba1460 to 127.0.0.1:63943 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 05:00:58,244 DEBUG [Listener at localhost/42165] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@72339fee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 05:00:58,247 DEBUG [hconnection-0x2ac17ebb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:58,248 INFO [RS-EventLoopGroup-4-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44480, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:58,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,460 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 05:00:58,463 DEBUG [hconnection-0x2ac17ebb-shared-pool-1] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:58,464 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59604, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:58,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,606 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 05:00:58,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,774 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 05:00:58,777 DEBUG [hconnection-0x2ac17ebb-shared-pool-3] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:00:58,778 INFO [RS-EventLoopGroup-5-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33542, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:00:58,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,941 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 05:00:58,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:58,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,089 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 05:00:59,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,611 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x03516549 to 127.0.0.1:63943 2023-02-13 05:00:59,611 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:00:59,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:00:59,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HMobStore(386): Fail to read the cell, the mob file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 doesn't exist java.io.FileNotFoundException: File does not exist: hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1527) at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1520) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:446) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:168) at org.apache.hadoop.hbase.regionserver.StoreFileInfo.(StoreFileInfo.java:130) at org.apache.hadoop.hbase.regionserver.HStoreFile.(HStoreFile.java:208) at org.apache.hadoop.hbase.mob.MobFile.create(MobFile.java:140) at org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:202) at org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:376) at org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:333) at org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:79) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:145) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:7197) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:7363) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:7133) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3352) at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:3618) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44996) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:00,882 INFO [Listener at localhost/42165] client.HBaseAdmin$15(890): Started disable of testRestoreSnapshot-1676264436786 2023-02-13 05:01:00,882 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testRestoreSnapshot-1676264436786 2023-02-13 05:01:00,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=188, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:01:00,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-13 05:01:00,886 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264460886"}]},"ts":"1676264460886"} 2023-02-13 05:01:00,888 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=DISABLING in hbase:meta 2023-02-13 05:01:00,889 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testRestoreSnapshot-1676264436786 to state=DISABLING 2023-02-13 05:01:00,890 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN}, {pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN}, {pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN}, {pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN}, {pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN}, {pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN}] 2023-02-13 05:01:00,891 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=193, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN 2023-02-13 05:01:00,892 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=191, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN 2023-02-13 05:01:00,892 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=194, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN 2023-02-13 05:01:00,892 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=192, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN 2023-02-13 05:01:00,892 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=190, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN 2023-02-13 05:01:00,892 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:00,892 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:00,892 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264460892"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264460892"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264460892"}]},"ts":"1676264460892"} 2023-02-13 05:01:00,893 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264460892"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264460892"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264460892"}]},"ts":"1676264460892"} 2023-02-13 05:01:00,893 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:00,893 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:00,893 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264460893"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264460893"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264460893"}]},"ts":"1676264460893"} 2023-02-13 05:01:00,893 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264460893"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264460893"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264460893"}]},"ts":"1676264460893"} 2023-02-13 05:01:00,894 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:00,894 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264460894"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264460894"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264460894"}]},"ts":"1676264460894"} 2023-02-13 05:01:00,894 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=195, ppid=193, state=RUNNABLE; CloseRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:01:00,895 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=196, ppid=191, state=RUNNABLE; CloseRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:01:00,895 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=197, ppid=194, state=RUNNABLE; CloseRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:01:00,899 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=198, ppid=192, state=RUNNABLE; CloseRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:01:00,899 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=199, ppid=190, state=RUNNABLE; CloseRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:01:00,899 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=189, ppid=188, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN 2023-02-13 05:01:00,900 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:00,900 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264460900"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264460900"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264460900"}]},"ts":"1676264460900"} 2023-02-13 05:01:00,901 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=200, ppid=189, state=RUNNABLE; CloseRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:01:01,046 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:01:01,047 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 347ea8719201fbf1ee1e731b16986087, disabling compactions & flushes 2023-02-13 05:01:01,048 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:01:01,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:01:01,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. after waiting 0 ms 2023-02-13 05:01:01,048 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:01:01,051 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:01:01,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d429d785f30801586e3db3ae3a9428b8, disabling compactions & flushes 2023-02-13 05:01:01,052 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:01:01,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:01:01,052 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:01:01,052 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. after waiting 0 ms 2023-02-13 05:01:01,053 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing d063b19c8c76cc04b197243b846f83ab, disabling compactions & flushes 2023-02-13 05:01:01,053 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:01:01,053 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:01:01,053 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:01:01,053 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. after waiting 0 ms 2023-02-13 05:01:01,053 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:01:01,060 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 05:01:01,061 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087. 2023-02-13 05:01:01,061 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 347ea8719201fbf1ee1e731b16986087: Waiting for close lock at 1676264461047Running coprocessor pre-close hooks at 1676264461047Disabling compacts and flushes for region at 1676264461047Disabling writes for close at 1676264461048 (+1 ms)Writing region close event to WAL at 1676264461051 (+3 ms)Running coprocessor post-close hooks at 1676264461061 (+10 ms)Closed at 1676264461061 2023-02-13 05:01:01,061 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 05:01:01,061 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 05:01:01,062 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8. 2023-02-13 05:01:01,062 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d429d785f30801586e3db3ae3a9428b8: Waiting for close lock at 1676264461052Running coprocessor pre-close hooks at 1676264461052Disabling compacts and flushes for region at 1676264461052Disabling writes for close at 1676264461053 (+1 ms)Writing region close event to WAL at 1676264461058 (+5 ms)Running coprocessor post-close hooks at 1676264461062 (+4 ms)Closed at 1676264461062 2023-02-13 05:01:01,062 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab. 2023-02-13 05:01:01,062 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for d063b19c8c76cc04b197243b846f83ab: Waiting for close lock at 1676264461052Running coprocessor pre-close hooks at 1676264461052Disabling compacts and flushes for region at 1676264461052Disabling writes for close at 1676264461053 (+1 ms)Writing region close event to WAL at 1676264461058 (+5 ms)Running coprocessor post-close hooks at 1676264461062 (+4 ms)Closed at 1676264461062 2023-02-13 05:01:01,062 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 347ea8719201fbf1ee1e731b16986087 2023-02-13 05:01:01,062 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:01:01,063 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ee2d32fb6918e03288e862be581d9bcb, disabling compactions & flushes 2023-02-13 05:01:01,063 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:01:01,063 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:01:01,063 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. after waiting 0 ms 2023-02-13 05:01:01,063 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:01:01,063 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=192 updating hbase:meta row=347ea8719201fbf1ee1e731b16986087, regionState=CLOSED 2023-02-13 05:01:01,063 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264461063"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264461063"}]},"ts":"1676264461063"} 2023-02-13 05:01:01,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:01:01,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:01:01,066 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 11ed57e84fcc1f7d47e9ad2ed09021f8, disabling compactions & flushes 2023-02-13 05:01:01,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:01:01,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:01:01,068 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=189 updating hbase:meta row=d429d785f30801586e3db3ae3a9428b8, regionState=CLOSED 2023-02-13 05:01:01,068 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. after waiting 0 ms 2023-02-13 05:01:01,071 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:01:01,071 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264461067"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264461067"}]},"ts":"1676264461067"} 2023-02-13 05:01:01,071 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:01:01,071 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:01:01,072 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=190 updating hbase:meta row=d063b19c8c76cc04b197243b846f83ab, regionState=CLOSED 2023-02-13 05:01:01,073 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264461072"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264461072"}]},"ts":"1676264461072"} 2023-02-13 05:01:01,073 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 7017e1f9205c2ebde9f23df4733114d5, disabling compactions & flushes 2023-02-13 05:01:01,073 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:01:01,073 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:01:01,073 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. after waiting 0 ms 2023-02-13 05:01:01,073 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:01:01,077 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=198, resume processing ppid=192 2023-02-13 05:01:01,077 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=198, ppid=192, state=SUCCESS; CloseRegionProcedure 347ea8719201fbf1ee1e731b16986087, server=jenkins-hbase4.apache.org,34363,1676264434089 in 176 msec 2023-02-13 05:01:01,078 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=200, resume processing ppid=189 2023-02-13 05:01:01,078 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=200, ppid=189, state=SUCCESS; CloseRegionProcedure d429d785f30801586e3db3ae3a9428b8, server=jenkins-hbase4.apache.org,39885,1676264434059 in 173 msec 2023-02-13 05:01:01,078 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 05:01:01,078 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 05:01:01,078 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=199, resume processing ppid=190 2023-02-13 05:01:01,078 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=192, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=347ea8719201fbf1ee1e731b16986087, UNASSIGN in 187 msec 2023-02-13 05:01:01,078 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb. 2023-02-13 05:01:01,078 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=199, ppid=190, state=SUCCESS; CloseRegionProcedure d063b19c8c76cc04b197243b846f83ab, server=jenkins-hbase4.apache.org,42589,1676264434015 in 176 msec 2023-02-13 05:01:01,078 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ee2d32fb6918e03288e862be581d9bcb: Waiting for close lock at 1676264461062Running coprocessor pre-close hooks at 1676264461062Disabling compacts and flushes for region at 1676264461062Disabling writes for close at 1676264461063 (+1 ms)Writing region close event to WAL at 1676264461071 (+8 ms)Running coprocessor post-close hooks at 1676264461078 (+7 ms)Closed at 1676264461078 2023-02-13 05:01:01,079 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8. 2023-02-13 05:01:01,079 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 11ed57e84fcc1f7d47e9ad2ed09021f8: Waiting for close lock at 1676264461065Running coprocessor pre-close hooks at 1676264461065Disabling compacts and flushes for region at 1676264461065Disabling writes for close at 1676264461071 (+6 ms)Writing region close event to WAL at 1676264461072 (+1 ms)Running coprocessor post-close hooks at 1676264461079 (+7 ms)Closed at 1676264461079 2023-02-13 05:01:01,080 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=189, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d429d785f30801586e3db3ae3a9428b8, UNASSIGN in 188 msec 2023-02-13 05:01:01,080 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:01:01,080 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=190, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=d063b19c8c76cc04b197243b846f83ab, UNASSIGN in 188 msec 2023-02-13 05:01:01,080 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=193 updating hbase:meta row=ee2d32fb6918e03288e862be581d9bcb, regionState=CLOSED 2023-02-13 05:01:01,080 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264461080"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264461080"}]},"ts":"1676264461080"} 2023-02-13 05:01:01,081 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:01:01,081 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=191 updating hbase:meta row=11ed57e84fcc1f7d47e9ad2ed09021f8, regionState=CLOSED 2023-02-13 05:01:01,081 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"regioninfo","vlen":69,"tag":[],"timestamp":"1676264461081"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264461081"}]},"ts":"1676264461081"} 2023-02-13 05:01:01,083 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=15 2023-02-13 05:01:01,083 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=195, resume processing ppid=193 2023-02-13 05:01:01,083 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=195, ppid=193, state=SUCCESS; CloseRegionProcedure ee2d32fb6918e03288e862be581d9bcb, server=jenkins-hbase4.apache.org,34363,1676264434089 in 187 msec 2023-02-13 05:01:01,083 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5. 2023-02-13 05:01:01,083 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 7017e1f9205c2ebde9f23df4733114d5: Waiting for close lock at 1676264461071Running coprocessor pre-close hooks at 1676264461071Disabling compacts and flushes for region at 1676264461071Disabling writes for close at 1676264461073 (+2 ms)Writing region close event to WAL at 1676264461074 (+1 ms)Running coprocessor post-close hooks at 1676264461083 (+9 ms)Closed at 1676264461083 2023-02-13 05:01:01,084 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=196, resume processing ppid=191 2023-02-13 05:01:01,084 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=193, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=ee2d32fb6918e03288e862be581d9bcb, UNASSIGN in 193 msec 2023-02-13 05:01:01,084 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=196, ppid=191, state=SUCCESS; CloseRegionProcedure 11ed57e84fcc1f7d47e9ad2ed09021f8, server=jenkins-hbase4.apache.org,39885,1676264434059 in 187 msec 2023-02-13 05:01:01,085 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:01:01,085 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=194 updating hbase:meta row=7017e1f9205c2ebde9f23df4733114d5, regionState=CLOSED 2023-02-13 05:01:01,085 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"regioninfo","vlen":68,"tag":[],"timestamp":"1676264461085"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264461085"}]},"ts":"1676264461085"} 2023-02-13 05:01:01,085 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=191, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=11ed57e84fcc1f7d47e9ad2ed09021f8, UNASSIGN in 194 msec 2023-02-13 05:01:01,087 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=197, resume processing ppid=194 2023-02-13 05:01:01,087 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=197, ppid=194, state=SUCCESS; CloseRegionProcedure 7017e1f9205c2ebde9f23df4733114d5, server=jenkins-hbase4.apache.org,42589,1676264434015 in 191 msec 2023-02-13 05:01:01,088 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=194, resume processing ppid=188 2023-02-13 05:01:01,088 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=194, ppid=188, state=SUCCESS; TransitRegionStateProcedure table=testRestoreSnapshot-1676264436786, region=7017e1f9205c2ebde9f23df4733114d5, UNASSIGN in 197 msec 2023-02-13 05:01:01,088 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264461088"}]},"ts":"1676264461088"} 2023-02-13 05:01:01,089 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testRestoreSnapshot-1676264436786, state=DISABLED in hbase:meta 2023-02-13 05:01:01,091 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testRestoreSnapshot-1676264436786 to state=DISABLED 2023-02-13 05:01:01,092 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=188, state=SUCCESS; DisableTableProcedure table=testRestoreSnapshot-1676264436786 in 209 msec 2023-02-13 05:01:01,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=188 2023-02-13 05:01:01,137 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testRestoreSnapshot-1676264436786, procId: 188 completed 2023-02-13 05:01:01,138 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testRestoreSnapshot-1676264436786 2023-02-13 05:01:01,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:01:01,140 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=201, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:01:01,140 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=201, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:01:01,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-13 05:01:01,147 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:01:01,147 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:01:01,147 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:01:01,147 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:01:01,147 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:01:01,147 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:01:01,148 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf, FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/recovered.edits] 2023-02-13 05:01:01,149 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf, FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/recovered.edits] 2023-02-13 05:01:01,149 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf, FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/recovered.edits] 2023-02-13 05:01:01,149 DEBUG [HFileArchiver-4] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf, FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/recovered.edits] 2023-02-13 05:01:01,149 DEBUG [HFileArchiver-8] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf, FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/recovered.edits] 2023-02-13 05:01:01,149 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf, FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/recovered.edits] 2023-02-13 05:01:01,152 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d15ef8effb63426f88cf28b863b418da already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d15ef8effb63426f88cf28b863b418da. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457216 currentMtime: 1676264457571 2023-02-13 05:01:01,153 ERROR [HFileArchiver-8] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-3ae4957471034eeb9ed26ec1a046807b already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-3ae4957471034eeb9ed26ec1a046807b. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457216 currentMtime: 1676264457570 2023-02-13 05:01:01,153 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-811ac31223674808858dc07f9153be03 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-811ac31223674808858dc07f9153be03. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457215 currentMtime: 1676264457571 2023-02-13 05:01:01,153 INFO [HFileArchiver-8] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-3ae4957471034eeb9ed26ec1a046807b to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-3ae4957471034eeb9ed26ec1a046807b.1676264461149. 2023-02-13 05:01:01,153 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-3f25ec8511e84234a9c7a722ca070d0b already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-3f25ec8511e84234a9c7a722ca070d0b. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457216 currentMtime: 1676264457570 2023-02-13 05:01:01,153 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d15ef8effb63426f88cf28b863b418da to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d15ef8effb63426f88cf28b863b418da.1676264461149. 2023-02-13 05:01:01,153 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-519efdc2aee8432c80befe61da0da81c already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-519efdc2aee8432c80befe61da0da81c. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457216 currentMtime: 1676264457572 2023-02-13 05:01:01,153 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-811ac31223674808858dc07f9153be03 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-811ac31223674808858dc07f9153be03.1676264461149. 2023-02-13 05:01:01,154 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-075fd0e105614d9aaf0995ac7325a2fb already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-075fd0e105614d9aaf0995ac7325a2fb. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457216 currentMtime: 1676264457570 2023-02-13 05:01:01,154 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-3f25ec8511e84234a9c7a722ca070d0b to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-3f25ec8511e84234a9c7a722ca070d0b.1676264461149. 2023-02-13 05:01:01,154 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-519efdc2aee8432c80befe61da0da81c to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-519efdc2aee8432c80befe61da0da81c.1676264461149. 2023-02-13 05:01:01,154 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d15ef8effb63426f88cf28b863b418da to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d15ef8effb63426f88cf28b863b418da 2023-02-13 05:01:01,154 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-075fd0e105614d9aaf0995ac7325a2fb to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-075fd0e105614d9aaf0995ac7325a2fb.1676264461149. 2023-02-13 05:01:01,154 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-3ae4957471034eeb9ed26ec1a046807b to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-3ae4957471034eeb9ed26ec1a046807b 2023-02-13 05:01:01,155 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-811ac31223674808858dc07f9153be03 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-811ac31223674808858dc07f9153be03 2023-02-13 05:01:01,155 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-3f25ec8511e84234a9c7a722ca070d0b to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-3f25ec8511e84234a9c7a722ca070d0b 2023-02-13 05:01:01,155 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-519efdc2aee8432c80befe61da0da81c to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-519efdc2aee8432c80befe61da0da81c 2023-02-13 05:01:01,156 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-075fd0e105614d9aaf0995ac7325a2fb to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-075fd0e105614d9aaf0995ac7325a2fb 2023-02-13 05:01:01,157 ERROR [HFileArchiver-8] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-6414549fb99340299c9cb4b42e13bd55 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-6414549fb99340299c9cb4b42e13bd55. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457218 currentMtime: 1676264457573 2023-02-13 05:01:01,157 ERROR [HFileArchiver-2] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d318209d74c44b3fb2d00363c5890672 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d318209d74c44b3fb2d00363c5890672. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457218 currentMtime: 1676264457575 2023-02-13 05:01:01,157 ERROR [HFileArchiver-4] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-b95f98eee2d241bea094905381835a03 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-b95f98eee2d241bea094905381835a03. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457218 currentMtime: 1676264457575 2023-02-13 05:01:01,157 ERROR [HFileArchiver-1] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-b0b32b9f22144056b97173b2a69002a5 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-b0b32b9f22144056b97173b2a69002a5. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457218 currentMtime: 1676264457575 2023-02-13 05:01:01,157 INFO [HFileArchiver-8] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-6414549fb99340299c9cb4b42e13bd55 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-6414549fb99340299c9cb4b42e13bd55.1676264461149. 2023-02-13 05:01:01,157 INFO [HFileArchiver-2] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d318209d74c44b3fb2d00363c5890672 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d318209d74c44b3fb2d00363c5890672.1676264461149. 2023-02-13 05:01:01,157 INFO [HFileArchiver-4] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-b95f98eee2d241bea094905381835a03 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-b95f98eee2d241bea094905381835a03.1676264461149. 2023-02-13 05:01:01,157 ERROR [HFileArchiver-3] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-5ba29d67350f40888be6bfe3fec7f441 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-5ba29d67350f40888be6bfe3fec7f441. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457218 currentMtime: 1676264457575 2023-02-13 05:01:01,157 INFO [HFileArchiver-1] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-b0b32b9f22144056b97173b2a69002a5 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-b0b32b9f22144056b97173b2a69002a5.1676264461149. 2023-02-13 05:01:01,158 INFO [HFileArchiver-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-5ba29d67350f40888be6bfe3fec7f441 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-5ba29d67350f40888be6bfe3fec7f441.1676264461149. 2023-02-13 05:01:01,158 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d318209d74c44b3fb2d00363c5890672 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/cf/testRestoreSnapshot-1676264436786=7017e1f9205c2ebde9f23df4733114d5-d318209d74c44b3fb2d00363c5890672 2023-02-13 05:01:01,158 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-6414549fb99340299c9cb4b42e13bd55 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/cf/testRestoreSnapshot-1676264436786=11ed57e84fcc1f7d47e9ad2ed09021f8-6414549fb99340299c9cb4b42e13bd55 2023-02-13 05:01:01,158 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-b95f98eee2d241bea094905381835a03 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/cf/testRestoreSnapshot-1676264436786=d429d785f30801586e3db3ae3a9428b8-b95f98eee2d241bea094905381835a03 2023-02-13 05:01:01,159 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-b0b32b9f22144056b97173b2a69002a5 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/cf/testRestoreSnapshot-1676264436786=347ea8719201fbf1ee1e731b16986087-b0b32b9f22144056b97173b2a69002a5 2023-02-13 05:01:01,159 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-5ba29d67350f40888be6bfe3fec7f441 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/cf/testRestoreSnapshot-1676264436786=d063b19c8c76cc04b197243b846f83ab-5ba29d67350f40888be6bfe3fec7f441 2023-02-13 05:01:01,160 ERROR [HFileArchiver-6] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-d3866b21c6944cf0882c65fec182cb40 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-d3866b21c6944cf0882c65fec182cb40. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457219 currentMtime: 1676264457574 2023-02-13 05:01:01,161 INFO [HFileArchiver-6] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-d3866b21c6944cf0882c65fec182cb40 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-d3866b21c6944cf0882c65fec182cb40.1676264461149. 2023-02-13 05:01:01,161 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/recovered.edits/18.seqid to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5/recovered.edits/18.seqid 2023-02-13 05:01:01,162 DEBUG [HFileArchiver-8] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/recovered.edits/18.seqid to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8/recovered.edits/18.seqid 2023-02-13 05:01:01,162 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/7017e1f9205c2ebde9f23df4733114d5 2023-02-13 05:01:01,162 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-d3866b21c6944cf0882c65fec182cb40 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/cf/testRestoreSnapshot-1676264436786=ee2d32fb6918e03288e862be581d9bcb-d3866b21c6944cf0882c65fec182cb40 2023-02-13 05:01:01,162 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/recovered.edits/18.seqid to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087/recovered.edits/18.seqid 2023-02-13 05:01:01,162 DEBUG [HFileArchiver-8] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/11ed57e84fcc1f7d47e9ad2ed09021f8 2023-02-13 05:01:01,163 DEBUG [HFileArchiver-4] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/recovered.edits/18.seqid to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8/recovered.edits/18.seqid 2023-02-13 05:01:01,163 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/347ea8719201fbf1ee1e731b16986087 2023-02-13 05:01:01,163 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/recovered.edits/18.seqid to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab/recovered.edits/18.seqid 2023-02-13 05:01:01,163 DEBUG [HFileArchiver-4] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d429d785f30801586e3db3ae3a9428b8 2023-02-13 05:01:01,164 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/d063b19c8c76cc04b197243b846f83ab 2023-02-13 05:01:01,165 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/recovered.edits/18.seqid to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb/recovered.edits/18.seqid 2023-02-13 05:01:01,165 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testRestoreSnapshot-1676264436786/ee2d32fb6918e03288e862be581d9bcb 2023-02-13 05:01:01,165 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived testRestoreSnapshot-1676264436786 regions 2023-02-13 05:01:01,166 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6 2023-02-13 05:01:01,166 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf] 2023-02-13 05:01:01,168 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457227 currentMtime: 1676264457501 2023-02-13 05:01:01,169 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0.1676264461166. 2023-02-13 05:01:01,169 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-a87ff679a2f3e71d9181a67b7542122c202302134e52bcbf6b0a42b0af9c136eb22bbef0 2023-02-13 05:01:01,170 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457228 currentMtime: 1676264457503 2023-02-13 05:01:01,170 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435.1676264461166. 2023-02-13 05:01:01,171 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-a87ff679a2f3e71d9181a67b7542122c202302138c2142bd212049fc938b1c5e6b986435 2023-02-13 05:01:01,172 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457229 currentMtime: 1676264457505 2023-02-13 05:01:01,172 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827.1676264461166. 2023-02-13 05:01:01,173 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c4ca4238a0b923820dcc509a6f75849b2023021378052df98393482699b23acdc56eb827 2023-02-13 05:01:01,174 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457230 currentMtime: 1676264457507 2023-02-13 05:01:01,174 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540.1676264461166. 2023-02-13 05:01:01,175 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c4ca4238a0b923820dcc509a6f75849b20230213ed990bb13464454cb86daf0853ef2540 2023-02-13 05:01:01,175 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457231 currentMtime: 1676264457509 2023-02-13 05:01:01,176 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c.1676264461166. 2023-02-13 05:01:01,177 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c81e728d9d4c2f636f067f89cc14862c2023021358d454c6e6ee41d0a214935d23d53c1c 2023-02-13 05:01:01,177 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457232 currentMtime: 1676264457511 2023-02-13 05:01:01,178 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6.1676264461166. 2023-02-13 05:01:01,178 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-c81e728d9d4c2f636f067f89cc14862c202302139882449212ef40f6afbe7eec7dcc59c6 2023-02-13 05:01:01,179 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457233 currentMtime: 1676264457513 2023-02-13 05:01:01,180 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2.1676264461166. 2023-02-13 05:01:01,180 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-d41d8cd98f00b204e9800998ecf8427e202302138ad33be89e5f421f9154223ef8febab2 2023-02-13 05:01:01,181 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457235 currentMtime: 1676264457515 2023-02-13 05:01:01,181 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509.1676264461166. 2023-02-13 05:01:01,182 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-d41d8cd98f00b204e9800998ecf8427e20230213acf7aae868184ccc98140bcc38b57509 2023-02-13 05:01:01,183 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457236 currentMtime: 1676264457517 2023-02-13 05:01:01,183 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3.1676264461166. 2023-02-13 05:01:01,184 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-e4da3b7fbbce2345d7772b0674a318d5202302135790ff793c174c908eb2869417b71cc3 2023-02-13 05:01:01,184 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457237 currentMtime: 1676264457518 2023-02-13 05:01:01,185 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b.1676264461166. 2023-02-13 05:01:01,186 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-e4da3b7fbbce2345d7772b0674a318d5202302138b9fb0adf2e24d6a8ae667c12eba1b1b 2023-02-13 05:01:01,186 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457238 currentMtime: 1676264457520 2023-02-13 05:01:01,187 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10.1676264461166. 2023-02-13 05:01:01,187 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-eccbc87e4b5ce2fe28308fd9f2a7baf32023021320afec8924c64dadb28b68f99dceeb10 2023-02-13 05:01:01,188 ERROR [PEWorker-3] backup.HFileArchiver(523): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 already exists in archive, moving to timestamped backup and overwriting current FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2. archiveLen: 0 currentLen: 0 archiveMtime: 1676264457239 currentMtime: 1676264457522 2023-02-13 05:01:01,188 INFO [PEWorker-3] backup.HFileArchiver(540): Backed up archive file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2.1676264461166. 2023-02-13 05:01:01,189 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6/cf/testRestoreSnapshot-1676264436786=93bce92e4aef17f80aeb287d234dc9c6-eccbc87e4b5ce2fe28308fd9f2a7baf32023021363378f1c825b4b3f9b77eaeb48e95cd2 2023-02-13 05:01:01,189 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testRestoreSnapshot-1676264436786/93bce92e4aef17f80aeb287d234dc9c6 2023-02-13 05:01:01,191 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=201, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:01:01,193 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testRestoreSnapshot-1676264436786 from hbase:meta 2023-02-13 05:01:01,195 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'testRestoreSnapshot-1676264436786' descriptor. 2023-02-13 05:01:01,196 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=201, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:01:01,196 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'testRestoreSnapshot-1676264436786' from region states. 2023-02-13 05:01:01,196 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264461196"}]},"ts":"9223372036854775807"} 2023-02-13 05:01:01,196 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264461196"}]},"ts":"9223372036854775807"} 2023-02-13 05:01:01,196 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264461196"}]},"ts":"9223372036854775807"} 2023-02-13 05:01:01,196 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264461196"}]},"ts":"9223372036854775807"} 2023-02-13 05:01:01,196 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264461196"}]},"ts":"9223372036854775807"} 2023-02-13 05:01:01,196 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264461196"}]},"ts":"9223372036854775807"} 2023-02-13 05:01:01,198 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 05:01:01,198 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d429d785f30801586e3db3ae3a9428b8, NAME => 'testRestoreSnapshot-1676264436786,,1676264436809.d429d785f30801586e3db3ae3a9428b8.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => d063b19c8c76cc04b197243b846f83ab, NAME => 'testRestoreSnapshot-1676264436786,1,1676264436809.d063b19c8c76cc04b197243b846f83ab.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => 11ed57e84fcc1f7d47e9ad2ed09021f8, NAME => 'testRestoreSnapshot-1676264436786,2,1676264436809.11ed57e84fcc1f7d47e9ad2ed09021f8.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 347ea8719201fbf1ee1e731b16986087, NAME => 'testRestoreSnapshot-1676264436786,3,1676264436809.347ea8719201fbf1ee1e731b16986087.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => ee2d32fb6918e03288e862be581d9bcb, NAME => 'testRestoreSnapshot-1676264436786,4,1676264436809.ee2d32fb6918e03288e862be581d9bcb.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => 7017e1f9205c2ebde9f23df4733114d5, NAME => 'testRestoreSnapshot-1676264436786,5,1676264436809.7017e1f9205c2ebde9f23df4733114d5.', STARTKEY => '5', ENDKEY => ''}] 2023-02-13 05:01:01,198 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'testRestoreSnapshot-1676264436786' as deleted. 2023-02-13 05:01:01,198 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testRestoreSnapshot-1676264436786","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676264461198"}]},"ts":"9223372036854775807"} 2023-02-13 05:01:01,199 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table testRestoreSnapshot-1676264436786 state from META 2023-02-13 05:01:01,201 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=201, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testRestoreSnapshot-1676264436786 2023-02-13 05:01:01,202 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=201, state=SUCCESS; DeleteTableProcedure table=testRestoreSnapshot-1676264436786 in 63 msec 2023-02-13 05:01:01,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=201 2023-02-13 05:01:01,392 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testRestoreSnapshot-1676264436786, procId: 201 completed 2023-02-13 05:01:01,398 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676264436786" 2023-02-13 05:01:01,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676264436786 2023-02-13 05:01:01,400 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676264436786" 2023-02-13 05:01:01,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676264436786 2023-02-13 05:01:01,402 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb1-1676264436786" 2023-02-13 05:01:01,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(333): Deleting snapshot: snaptb1-1676264436786 2023-02-13 05:01:01,431 INFO [Listener at localhost/42165] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testRestoreSnapshot Thread=428 (was 360) Potentially hanging thread: hconnection-0x10a3a304-shared-pool-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x10a3a304-shared-pool-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2ac17ebb-shared-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1515929686_15 at /127.0.0.1:60048 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/cluster_911aa46c-724d-9809-31e5-34f51d4314f6/dfs/data/data6/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-889939022_15 at /127.0.0.1:55488 [Waiting for operation #6] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1515929686_15 at /127.0.0.1:43178 [Waiting for operation #568] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x10a3a304-shared-pool-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_2104806991_15 at /127.0.0.1:60072 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2ac17ebb-metaLookup-shared--pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/cluster_911aa46c-724d-9809-31e5-34f51d4314f6/dfs/data/data1/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-2 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2ac17ebb-shared-pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-3-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1515929686_15 at /127.0.0.1:60096 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-15 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_200456204_15 at /127.0.0.1:59968 [Waiting for operation #479] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-889939022_15 at /127.0.0.1:46872 [Waiting for operation #1361] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2ac17ebb-shared-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-889939022_15 at /127.0.0.1:38708 [Waiting for operation #812] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x315dbdb4-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_2104806991_15 at /127.0.0.1:55502 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_200456204_15 at /127.0.0.1:59982 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2ac17ebb-shared-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2ac17ebb-shared-pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_200456204_15 at /127.0.0.1:60068 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/cluster_911aa46c-724d-9809-31e5-34f51d4314f6/dfs/data/data5/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1515929686_15 at /127.0.0.1:36234 [Waiting for operation #1028] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-889939022_15 at /127.0.0.1:55498 [Waiting for operation #5] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1515929686_15 at /127.0.0.1:60088 [Waiting for operation #2] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-16 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/cluster_911aa46c-724d-9809-31e5-34f51d4314f6/dfs/data/data4/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2ac17ebb-metaLookup-shared--pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/cluster_911aa46c-724d-9809-31e5-34f51d4314f6/dfs/data/data3/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x10a3a304-shared-pool-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63943@0x6eba1460-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506) Potentially hanging thread: MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63943@0x6eba1460 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.DelayQueue.poll(DelayQueue.java:259) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:324) org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$45/1518323605.run(Unknown Source) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2ac17ebb-shared-pool-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-889939022_15 at /127.0.0.1:55506 [Waiting for operation #512] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: htable-pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2ac17ebb-metaLookup-shared--pool-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2ac17ebb-metaLookup-shared--pool-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2ac17ebb-metaLookup-shared--pool-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: ReadOnlyZKClient-127.0.0.1:63943@0x6eba1460-SendThread(127.0.0.1:63943) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-889939022_15 at /127.0.0.1:60052 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Async disk worker #0 for volume /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/cluster_911aa46c-724d-9809-31e5-34f51d4314f6/dfs/data/data2/current sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-6-14 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=823 (was 697) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=168 (was 178), ProcessCount=168 (was 168), AvailableMemoryMB=3227 (was 3666) 2023-02-13 05:01:01,448 INFO [Listener at localhost/42165] hbase.ResourceChecker(147): before: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=428, OpenFileDescriptor=823, MaxFileDescriptor=60000, SystemLoadAverage=168, ProcessCount=168, AvailableMemoryMB=3226 2023-02-13 05:01:01,451 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$4(2102): Client=jenkins//172.31.14.131 create 'testCorruptedSnapshot-1676264461448', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 2023-02-13 05:01:01,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=testCorruptedSnapshot-1676264461448 2023-02-13 05:01:01,454 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676264461448 execute state=CREATE_TABLE_PRE_OPERATION 2023-02-13 05:01:01,454 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "testCorruptedSnapshot-1676264461448" procId is: 202 2023-02-13 05:01:01,455 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676264461448 execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-02-13 05:01:01,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-13 05:01:01,459 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:01,459 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:01,459 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:01,459 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:01,459 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:01,459 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8 2023-02-13 05:01:01,460 DEBUG [HFileArchiver-5] backup.HFileArchiver(153): Directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d empty. 2023-02-13 05:01:01,460 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2 empty. 2023-02-13 05:01:01,460 DEBUG [HFileArchiver-7] backup.HFileArchiver(153): Directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8 empty. 2023-02-13 05:01:01,460 DEBUG [HFileArchiver-5] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:01,460 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:01,461 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9 empty. 2023-02-13 05:01:01,461 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49 empty. 2023-02-13 05:01:01,461 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386 empty. 2023-02-13 05:01:01,461 DEBUG [HFileArchiver-7] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8 2023-02-13 05:01:01,461 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:01,461 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:01,461 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:01,461 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676264461448 regions 2023-02-13 05:01:01,476 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/.tabledesc/.tableinfo.0000000001 2023-02-13 05:01:01,478 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-0] regionserver.HRegion(7671): creating {ENCODED => 6b4351d6197a4df3ac583e92c336c18d, NAME => 'testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.', STARTKEY => '', ENDKEY => '1'}, tableDescriptor='testCorruptedSnapshot-1676264461448', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp 2023-02-13 05:01:01,478 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-1] regionserver.HRegion(7671): creating {ENCODED => ded68bdbede054e91359e396257503f8, NAME => 'testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.', STARTKEY => '1', ENDKEY => '2'}, tableDescriptor='testCorruptedSnapshot-1676264461448', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp 2023-02-13 05:01:01,478 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-2] regionserver.HRegion(7671): creating {ENCODED => c6ef1845e0de3d153503ed9c1b81ead2, NAME => 'testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.', STARTKEY => '2', ENDKEY => '3'}, tableDescriptor='testCorruptedSnapshot-1676264461448', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp 2023-02-13 05:01:01,487 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:01,487 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-0] regionserver.HRegion(1603): Closing 6b4351d6197a4df3ac583e92c336c18d, disabling compactions & flushes 2023-02-13 05:01:01,487 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:01,487 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:01,487 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. after waiting 0 ms 2023-02-13 05:01:01,487 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:01,487 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:01,487 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-0] regionserver.HRegion(1557): Region close journal for 6b4351d6197a4df3ac583e92c336c18d: Waiting for close lock at 1676264461487Disabling compacts and flushes for region at 1676264461487Disabling writes for close at 1676264461487Writing region close event to WAL at 1676264461487Closed at 1676264461487 2023-02-13 05:01:01,487 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-0] regionserver.HRegion(7671): creating {ENCODED => 3a6771e102cb35cd838b546f1db7ee49, NAME => 'testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.', STARTKEY => '3', ENDKEY => '4'}, tableDescriptor='testCorruptedSnapshot-1676264461448', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp 2023-02-13 05:01:01,488 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:01,488 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:01,488 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-2] regionserver.HRegion(1603): Closing c6ef1845e0de3d153503ed9c1b81ead2, disabling compactions & flushes 2023-02-13 05:01:01,488 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-1] regionserver.HRegion(1603): Closing ded68bdbede054e91359e396257503f8, disabling compactions & flushes 2023-02-13 05:01:01,488 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:01,489 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:01,489 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:01,489 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. after waiting 0 ms 2023-02-13 05:01:01,489 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:01,489 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:01,489 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:01,489 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. after waiting 0 ms 2023-02-13 05:01:01,489 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-2] regionserver.HRegion(1557): Region close journal for c6ef1845e0de3d153503ed9c1b81ead2: Waiting for close lock at 1676264461488Disabling compacts and flushes for region at 1676264461488Disabling writes for close at 1676264461489 (+1 ms)Writing region close event to WAL at 1676264461489Closed at 1676264461489 2023-02-13 05:01:01,489 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:01,489 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:01,489 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-1] regionserver.HRegion(1557): Region close journal for ded68bdbede054e91359e396257503f8: Waiting for close lock at 1676264461488Disabling compacts and flushes for region at 1676264461488Disabling writes for close at 1676264461489 (+1 ms)Writing region close event to WAL at 1676264461489Closed at 1676264461489 2023-02-13 05:01:01,489 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-2] regionserver.HRegion(7671): creating {ENCODED => f11a86da37311510c40c261c7bcedac9, NAME => 'testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.', STARTKEY => '4', ENDKEY => '5'}, tableDescriptor='testCorruptedSnapshot-1676264461448', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp 2023-02-13 05:01:01,490 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-1] regionserver.HRegion(7671): creating {ENCODED => ea5c27dda854cd4930a8099c7b035386, NAME => 'testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.', STARTKEY => '5', ENDKEY => ''}, tableDescriptor='testCorruptedSnapshot-1676264461448', {TABLE_ATTRIBUTES => {REGION_REPLICATION => '1'}}, {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, regionDir=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp 2023-02-13 05:01:01,504 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:01,505 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-0] regionserver.HRegion(1603): Closing 3a6771e102cb35cd838b546f1db7ee49, disabling compactions & flushes 2023-02-13 05:01:01,505 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:01,505 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:01,505 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. after waiting 0 ms 2023-02-13 05:01:01,505 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:01,505 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:01,505 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-0] regionserver.HRegion(1557): Region close journal for 3a6771e102cb35cd838b546f1db7ee49: Waiting for close lock at 1676264461505Disabling compacts and flushes for region at 1676264461505Disabling writes for close at 1676264461505Writing region close event to WAL at 1676264461505Closed at 1676264461505 2023-02-13 05:01:01,511 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-1] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:01,511 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-1] regionserver.HRegion(1603): Closing ea5c27dda854cd4930a8099c7b035386, disabling compactions & flushes 2023-02-13 05:01:01,511 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-1] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:01,511 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-1] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:01,511 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-1] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. after waiting 0 ms 2023-02-13 05:01:01,511 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-1] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:01,511 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-1] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:01,511 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-1] regionserver.HRegion(1557): Region close journal for ea5c27dda854cd4930a8099c7b035386: Waiting for close lock at 1676264461511Disabling compacts and flushes for region at 1676264461511Disabling writes for close at 1676264461511Writing region close event to WAL at 1676264461511Closed at 1676264461511 2023-02-13 05:01:01,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-13 05:01:01,905 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-2] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:01,905 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-2] regionserver.HRegion(1603): Closing f11a86da37311510c40c261c7bcedac9, disabling compactions & flushes 2023-02-13 05:01:01,905 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-2] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:01,905 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-2] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:01,905 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-2] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. after waiting 0 ms 2023-02-13 05:01:01,906 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-2] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:01,906 INFO [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-2] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:01,906 DEBUG [RegionOpenAndInit-testCorruptedSnapshot-1676264461448-pool-2] regionserver.HRegion(1557): Region close journal for f11a86da37311510c40c261c7bcedac9: Waiting for close lock at 1676264461905Disabling compacts and flushes for region at 1676264461905Disabling writes for close at 1676264461906 (+1 ms)Writing region close event to WAL at 1676264461906Closed at 1676264461906 2023-02-13 05:01:01,908 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676264461448 execute state=CREATE_TABLE_ADD_TO_META 2023-02-13 05:01:01,909 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264461909"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264461909"}]},"ts":"1676264461909"} 2023-02-13 05:01:01,909 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264461909"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264461909"}]},"ts":"1676264461909"} 2023-02-13 05:01:01,909 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264461909"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264461909"}]},"ts":"1676264461909"} 2023-02-13 05:01:01,909 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264461909"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264461909"}]},"ts":"1676264461909"} 2023-02-13 05:01:01,909 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264461909"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264461909"}]},"ts":"1676264461909"} 2023-02-13 05:01:01,909 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264461909"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264461909"}]},"ts":"1676264461909"} 2023-02-13 05:01:01,913 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 6 regions to meta. 2023-02-13 05:01:01,914 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676264461448 execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-02-13 05:01:01,914 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676264461448","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264461914"}]},"ts":"1676264461914"} 2023-02-13 05:01:01,915 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676264461448, state=ENABLING in hbase:meta 2023-02-13 05:01:01,920 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(334): Hosts are {jenkins-hbase4.apache.org=0} racks are {/default-rack=0} 2023-02-13 05:01:01,923 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0 2023-02-13 05:01:01,923 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 0 2023-02-13 05:01:01,923 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 0 2023-02-13 05:01:01,923 DEBUG [PEWorker-5] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=1, number of racks=1 2023-02-13 05:01:01,923 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, ASSIGN}, {pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, ASSIGN}, {pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, ASSIGN}, {pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, ASSIGN}, {pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, ASSIGN}, {pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, ASSIGN}] 2023-02-13 05:01:01,925 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, ASSIGN 2023-02-13 05:01:01,925 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, ASSIGN 2023-02-13 05:01:01,925 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, ASSIGN 2023-02-13 05:01:01,926 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, ASSIGN 2023-02-13 05:01:01,926 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=206, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39885,1676264434059; forceNewPlan=false, retain=false 2023-02-13 05:01:01,926 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=205, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42589,1676264434015; forceNewPlan=false, retain=false 2023-02-13 05:01:01,926 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=204, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34363,1676264434089; forceNewPlan=false, retain=false 2023-02-13 05:01:01,926 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=207, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,34363,1676264434089; forceNewPlan=false, retain=false 2023-02-13 05:01:01,926 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, ASSIGN 2023-02-13 05:01:01,927 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, ASSIGN 2023-02-13 05:01:01,928 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=208, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42589,1676264434015; forceNewPlan=false, retain=false 2023-02-13 05:01:01,928 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=203, ppid=202, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,39885,1676264434059; forceNewPlan=false, retain=false 2023-02-13 05:01:02,077 INFO [jenkins-hbase4:45905] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 05:01:02,080 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=c6ef1845e0de3d153503ed9c1b81ead2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:02,080 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=3a6771e102cb35cd838b546f1db7ee49, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:02,080 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264462080"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264462080"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264462080"}]},"ts":"1676264462080"} 2023-02-13 05:01:02,080 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=ded68bdbede054e91359e396257503f8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:02,080 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=6b4351d6197a4df3ac583e92c336c18d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:02,080 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=ea5c27dda854cd4930a8099c7b035386, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:02,081 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264462080"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264462080"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264462080"}]},"ts":"1676264462080"} 2023-02-13 05:01:02,081 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264462080"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264462080"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264462080"}]},"ts":"1676264462080"} 2023-02-13 05:01:02,081 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264462080"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264462080"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264462080"}]},"ts":"1676264462080"} 2023-02-13 05:01:02,080 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264462080"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264462080"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264462080"}]},"ts":"1676264462080"} 2023-02-13 05:01:02,082 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=209, ppid=205, state=RUNNABLE; OpenRegionProcedure c6ef1845e0de3d153503ed9c1b81ead2, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:01:02,082 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=210, ppid=203, state=RUNNABLE; OpenRegionProcedure 6b4351d6197a4df3ac583e92c336c18d, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:01:02,083 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=211, ppid=208, state=RUNNABLE; OpenRegionProcedure ea5c27dda854cd4930a8099c7b035386, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:01:02,083 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=212, ppid=204, state=RUNNABLE; OpenRegionProcedure ded68bdbede054e91359e396257503f8, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:01:02,084 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=213, ppid=206, state=RUNNABLE; OpenRegionProcedure 3a6771e102cb35cd838b546f1db7ee49, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:01:02,084 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=f11a86da37311510c40c261c7bcedac9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:02,084 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264462084"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264462084"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264462084"}]},"ts":"1676264462084"} 2023-02-13 05:01:02,085 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=214, ppid=207, state=RUNNABLE; OpenRegionProcedure f11a86da37311510c40c261c7bcedac9, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:01:02,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-13 05:01:02,235 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:02,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ea5c27dda854cd4930a8099c7b035386, NAME => 'testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 05:01:02,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676264461448 ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:02,236 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:02,236 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:02,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:02,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6b4351d6197a4df3ac583e92c336c18d, NAME => 'testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 05:01:02,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ded68bdbede054e91359e396257503f8, NAME => 'testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 05:01:02,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:02,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:02,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676264461448 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:02,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:02,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:02,236 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676264461448 ded68bdbede054e91359e396257503f8 2023-02-13 05:01:02,237 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:02,237 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:02,237 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ded68bdbede054e91359e396257503f8 2023-02-13 05:01:02,237 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ded68bdbede054e91359e396257503f8 2023-02-13 05:01:02,237 INFO [StoreOpener-ea5c27dda854cd4930a8099c7b035386-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:02,238 INFO [StoreOpener-6b4351d6197a4df3ac583e92c336c18d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:02,238 INFO [StoreOpener-ded68bdbede054e91359e396257503f8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ded68bdbede054e91359e396257503f8 2023-02-13 05:01:02,239 DEBUG [StoreOpener-ea5c27dda854cd4930a8099c7b035386-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/cf 2023-02-13 05:01:02,239 DEBUG [StoreOpener-ea5c27dda854cd4930a8099c7b035386-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/cf 2023-02-13 05:01:02,239 DEBUG [StoreOpener-6b4351d6197a4df3ac583e92c336c18d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/cf 2023-02-13 05:01:02,239 DEBUG [StoreOpener-6b4351d6197a4df3ac583e92c336c18d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/cf 2023-02-13 05:01:02,239 DEBUG [StoreOpener-ded68bdbede054e91359e396257503f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/cf 2023-02-13 05:01:02,239 DEBUG [StoreOpener-ded68bdbede054e91359e396257503f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/cf 2023-02-13 05:01:02,239 INFO [StoreOpener-ea5c27dda854cd4930a8099c7b035386-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ea5c27dda854cd4930a8099c7b035386 columnFamilyName cf 2023-02-13 05:01:02,239 INFO [StoreOpener-6b4351d6197a4df3ac583e92c336c18d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6b4351d6197a4df3ac583e92c336c18d columnFamilyName cf 2023-02-13 05:01:02,239 INFO [StoreOpener-ded68bdbede054e91359e396257503f8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ded68bdbede054e91359e396257503f8 columnFamilyName cf 2023-02-13 05:01:02,240 INFO [StoreOpener-ded68bdbede054e91359e396257503f8-1] regionserver.HStore(310): Store=ded68bdbede054e91359e396257503f8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:01:02,240 INFO [StoreOpener-6b4351d6197a4df3ac583e92c336c18d-1] regionserver.HStore(310): Store=6b4351d6197a4df3ac583e92c336c18d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:01:02,240 INFO [StoreOpener-ea5c27dda854cd4930a8099c7b035386-1] regionserver.HStore(310): Store=ea5c27dda854cd4930a8099c7b035386/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:01:02,241 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:02,241 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8 2023-02-13 05:01:02,241 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:02,241 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:02,241 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8 2023-02-13 05:01:02,242 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:02,244 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:02,244 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ded68bdbede054e91359e396257503f8 2023-02-13 05:01:02,244 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:02,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 05:01:02,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 05:01:02,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 05:01:02,246 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ea5c27dda854cd4930a8099c7b035386; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63339399, jitterRate=-0.056169405579566956}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:01:02,246 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ded68bdbede054e91359e396257503f8; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73987819, jitterRate=0.1025044173002243}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:01:02,246 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 6b4351d6197a4df3ac583e92c336c18d; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61327636, jitterRate=-0.0861470103263855}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:01:02,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ded68bdbede054e91359e396257503f8: Running coprocessor pre-open hook at 1676264462237Writing region info on filesystem at 1676264462237Initializing all the Stores at 1676264462237Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264462237Cleaning up temporary data from old regions at 1676264462242 (+5 ms)Cleaning up detritus from prior splits at 1676264462242Running coprocessor post-open hooks at 1676264462246 (+4 ms)Region opened successfully at 1676264462246 2023-02-13 05:01:02,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ea5c27dda854cd4930a8099c7b035386: Running coprocessor pre-open hook at 1676264462236Writing region info on filesystem at 1676264462236Initializing all the Stores at 1676264462237 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264462237Cleaning up temporary data from old regions at 1676264462242 (+5 ms)Cleaning up detritus from prior splits at 1676264462242Running coprocessor post-open hooks at 1676264462246 (+4 ms)Region opened successfully at 1676264462246 2023-02-13 05:01:02,246 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 6b4351d6197a4df3ac583e92c336c18d: Running coprocessor pre-open hook at 1676264462237Writing region info on filesystem at 1676264462237Initializing all the Stores at 1676264462237Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264462237Cleaning up temporary data from old regions at 1676264462242 (+5 ms)Cleaning up detritus from prior splits at 1676264462242Running coprocessor post-open hooks at 1676264462246 (+4 ms)Region opened successfully at 1676264462246 2023-02-13 05:01:02,247 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386., pid=211, masterSystemTime=1676264462233 2023-02-13 05:01:02,247 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d., pid=210, masterSystemTime=1676264462233 2023-02-13 05:01:02,247 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8., pid=212, masterSystemTime=1676264462234 2023-02-13 05:01:02,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:02,250 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:02,250 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:02,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c6ef1845e0de3d153503ed9c1b81ead2, NAME => 'testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 05:01:02,251 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=208 updating hbase:meta row=ea5c27dda854cd4930a8099c7b035386, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:02,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676264461448 c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:02,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:02,251 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264462251"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264462251"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264462251"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264462251"}]},"ts":"1676264462251"} 2023-02-13 05:01:02,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:02,251 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:02,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:02,251 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:02,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:02,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 3a6771e102cb35cd838b546f1db7ee49, NAME => 'testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 05:01:02,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676264461448 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:02,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:02,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:02,252 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:02,253 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:02,253 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=203 updating hbase:meta row=6b4351d6197a4df3ac583e92c336c18d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:02,253 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:02,253 INFO [StoreOpener-c6ef1845e0de3d153503ed9c1b81ead2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:02,253 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:02,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f11a86da37311510c40c261c7bcedac9, NAME => 'testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 05:01:02,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676264461448 f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:02,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:02,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:02,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:02,254 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=204 updating hbase:meta row=ded68bdbede054e91359e396257503f8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:02,254 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264462254"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264462254"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264462254"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264462254"}]},"ts":"1676264462254"} 2023-02-13 05:01:02,253 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264462253"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264462253"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264462253"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264462253"}]},"ts":"1676264462253"} 2023-02-13 05:01:02,254 INFO [StoreOpener-3a6771e102cb35cd838b546f1db7ee49-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:02,255 INFO [StoreOpener-f11a86da37311510c40c261c7bcedac9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:02,255 DEBUG [StoreOpener-c6ef1845e0de3d153503ed9c1b81ead2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/cf 2023-02-13 05:01:02,256 DEBUG [StoreOpener-c6ef1845e0de3d153503ed9c1b81ead2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/cf 2023-02-13 05:01:02,256 DEBUG [StoreOpener-3a6771e102cb35cd838b546f1db7ee49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/cf 2023-02-13 05:01:02,256 DEBUG [StoreOpener-3a6771e102cb35cd838b546f1db7ee49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/cf 2023-02-13 05:01:02,256 INFO [StoreOpener-c6ef1845e0de3d153503ed9c1b81ead2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c6ef1845e0de3d153503ed9c1b81ead2 columnFamilyName cf 2023-02-13 05:01:02,256 INFO [StoreOpener-3a6771e102cb35cd838b546f1db7ee49-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3a6771e102cb35cd838b546f1db7ee49 columnFamilyName cf 2023-02-13 05:01:02,257 DEBUG [StoreOpener-f11a86da37311510c40c261c7bcedac9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/cf 2023-02-13 05:01:02,257 DEBUG [StoreOpener-f11a86da37311510c40c261c7bcedac9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/cf 2023-02-13 05:01:02,257 INFO [StoreOpener-f11a86da37311510c40c261c7bcedac9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f11a86da37311510c40c261c7bcedac9 columnFamilyName cf 2023-02-13 05:01:02,257 INFO [StoreOpener-3a6771e102cb35cd838b546f1db7ee49-1] regionserver.HStore(310): Store=3a6771e102cb35cd838b546f1db7ee49/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:01:02,257 INFO [StoreOpener-c6ef1845e0de3d153503ed9c1b81ead2-1] regionserver.HStore(310): Store=c6ef1845e0de3d153503ed9c1b81ead2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:01:02,258 INFO [StoreOpener-f11a86da37311510c40c261c7bcedac9-1] regionserver.HStore(310): Store=f11a86da37311510c40c261c7bcedac9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:01:02,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:02,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:02,258 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:02,259 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=211, resume processing ppid=208 2023-02-13 05:01:02,259 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=211, ppid=208, state=SUCCESS; OpenRegionProcedure ea5c27dda854cd4930a8099c7b035386, server=jenkins-hbase4.apache.org,42589,1676264434015 in 171 msec 2023-02-13 05:01:02,259 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:02,261 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=212, resume processing ppid=204 2023-02-13 05:01:02,261 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=212, ppid=204, state=SUCCESS; OpenRegionProcedure ded68bdbede054e91359e396257503f8, server=jenkins-hbase4.apache.org,34363,1676264434089 in 174 msec 2023-02-13 05:01:02,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:02,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:02,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:02,262 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=208, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, ASSIGN in 336 msec 2023-02-13 05:01:02,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:02,264 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=204, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, ASSIGN in 338 msec 2023-02-13 05:01:02,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:02,268 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=210, resume processing ppid=203 2023-02-13 05:01:02,268 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=210, ppid=203, state=SUCCESS; OpenRegionProcedure 6b4351d6197a4df3ac583e92c336c18d, server=jenkins-hbase4.apache.org,39885,1676264434059 in 178 msec 2023-02-13 05:01:02,270 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=203, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, ASSIGN in 345 msec 2023-02-13 05:01:02,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 05:01:02,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 05:01:02,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-02-13 05:01:02,272 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 3a6771e102cb35cd838b546f1db7ee49; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71772896, jitterRate=0.06949949264526367}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:01:02,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 3a6771e102cb35cd838b546f1db7ee49: Running coprocessor pre-open hook at 1676264462252Writing region info on filesystem at 1676264462252Initializing all the Stores at 1676264462253 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264462253Cleaning up temporary data from old regions at 1676264462259 (+6 ms)Cleaning up detritus from prior splits at 1676264462259Running coprocessor post-open hooks at 1676264462272 (+13 ms)Region opened successfully at 1676264462272 2023-02-13 05:01:02,273 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c6ef1845e0de3d153503ed9c1b81ead2; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72253428, jitterRate=0.07665997743606567}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:01:02,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c6ef1845e0de3d153503ed9c1b81ead2: Running coprocessor pre-open hook at 1676264462251Writing region info on filesystem at 1676264462251Initializing all the Stores at 1676264462253 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264462253Cleaning up temporary data from old regions at 1676264462259 (+6 ms)Cleaning up detritus from prior splits at 1676264462260 (+1 ms)Running coprocessor post-open hooks at 1676264462273 (+13 ms)Region opened successfully at 1676264462273 2023-02-13 05:01:02,273 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f11a86da37311510c40c261c7bcedac9; next sequenceid=2; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64794816, jitterRate=-0.03448200225830078}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:01:02,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f11a86da37311510c40c261c7bcedac9: Running coprocessor pre-open hook at 1676264462254Writing region info on filesystem at 1676264462254Initializing all the Stores at 1676264462255 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264462255Cleaning up temporary data from old regions at 1676264462262 (+7 ms)Cleaning up detritus from prior splits at 1676264462263 (+1 ms)Running coprocessor post-open hooks at 1676264462273 (+10 ms)Region opened successfully at 1676264462273 2023-02-13 05:01:02,273 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49., pid=213, masterSystemTime=1676264462233 2023-02-13 05:01:02,274 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2., pid=209, masterSystemTime=1676264462233 2023-02-13 05:01:02,274 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9., pid=214, masterSystemTime=1676264462234 2023-02-13 05:01:02,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:02,274 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:02,275 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=206 updating hbase:meta row=3a6771e102cb35cd838b546f1db7ee49, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:02,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:02,275 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264462275"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264462275"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264462275"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264462275"}]},"ts":"1676264462275"} 2023-02-13 05:01:02,275 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:02,275 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=205 updating hbase:meta row=c6ef1845e0de3d153503ed9c1b81ead2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:02,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:02,276 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:02,276 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264462275"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264462275"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264462275"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264462275"}]},"ts":"1676264462275"} 2023-02-13 05:01:02,277 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=207 updating hbase:meta row=f11a86da37311510c40c261c7bcedac9, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:02,277 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264462277"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264462277"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264462277"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264462277"}]},"ts":"1676264462277"} 2023-02-13 05:01:02,279 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=213, resume processing ppid=206 2023-02-13 05:01:02,279 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=213, ppid=206, state=SUCCESS; OpenRegionProcedure 3a6771e102cb35cd838b546f1db7ee49, server=jenkins-hbase4.apache.org,39885,1676264434059 in 192 msec 2023-02-13 05:01:02,279 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=209, resume processing ppid=205 2023-02-13 05:01:02,279 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=209, ppid=205, state=SUCCESS; OpenRegionProcedure c6ef1845e0de3d153503ed9c1b81ead2, server=jenkins-hbase4.apache.org,42589,1676264434015 in 195 msec 2023-02-13 05:01:02,280 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=214, resume processing ppid=207 2023-02-13 05:01:02,280 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=206, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, ASSIGN in 356 msec 2023-02-13 05:01:02,280 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=214, ppid=207, state=SUCCESS; OpenRegionProcedure f11a86da37311510c40c261c7bcedac9, server=jenkins-hbase4.apache.org,34363,1676264434089 in 193 msec 2023-02-13 05:01:02,280 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=205, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, ASSIGN in 356 msec 2023-02-13 05:01:02,281 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=207, resume processing ppid=202 2023-02-13 05:01:02,282 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=207, ppid=202, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, ASSIGN in 357 msec 2023-02-13 05:01:02,286 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676264461448 execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-02-13 05:01:02,286 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676264461448","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264462286"}]},"ts":"1676264462286"} 2023-02-13 05:01:02,287 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676264461448, state=ENABLED in hbase:meta 2023-02-13 05:01:02,295 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=202, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=testCorruptedSnapshot-1676264461448 execute state=CREATE_TABLE_POST_OPERATION 2023-02-13 05:01:02,296 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=202, state=SUCCESS; CreateTableProcedure table=testCorruptedSnapshot-1676264461448 in 844 msec 2023-02-13 05:01:02,613 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 05:01:02,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=202 2023-02-13 05:01:02,958 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:testCorruptedSnapshot-1676264461448, procId: 202 completed 2023-02-13 05:01:02,961 DEBUG [Listener at localhost/42165] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676264461448 2023-02-13 05:01:02,961 DEBUG [Listener at localhost/42165] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:02,964 INFO [Listener at localhost/42165] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 05:01:02,976 INFO [Listener at localhost/42165] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676264461448 2023-02-13 05:01:02,976 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676264461448 2023-02-13 05:01:02,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=215, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676264461448 2023-02-13 05:01:02,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-13 05:01:02,980 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676264461448","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264462980"}]},"ts":"1676264462980"} 2023-02-13 05:01:02,981 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676264461448, state=DISABLING in hbase:meta 2023-02-13 05:01:02,983 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676264461448 to state=DISABLING 2023-02-13 05:01:02,984 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, UNASSIGN}, {pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, UNASSIGN}, {pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, UNASSIGN}, {pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, UNASSIGN}, {pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, UNASSIGN}, {pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, UNASSIGN}] 2023-02-13 05:01:02,985 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=216, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, UNASSIGN 2023-02-13 05:01:02,985 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=217, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, UNASSIGN 2023-02-13 05:01:02,986 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=219, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, UNASSIGN 2023-02-13 05:01:02,986 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=218, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, UNASSIGN 2023-02-13 05:01:02,986 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=221, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, UNASSIGN 2023-02-13 05:01:02,986 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=6b4351d6197a4df3ac583e92c336c18d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:02,986 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=ded68bdbede054e91359e396257503f8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:02,986 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=3a6771e102cb35cd838b546f1db7ee49, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:02,986 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264462986"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264462986"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264462986"}]},"ts":"1676264462986"} 2023-02-13 05:01:02,986 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264462986"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264462986"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264462986"}]},"ts":"1676264462986"} 2023-02-13 05:01:02,986 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=ea5c27dda854cd4930a8099c7b035386, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:02,986 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=c6ef1845e0de3d153503ed9c1b81ead2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:02,987 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264462986"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264462986"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264462986"}]},"ts":"1676264462986"} 2023-02-13 05:01:02,987 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264462986"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264462986"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264462986"}]},"ts":"1676264462986"} 2023-02-13 05:01:02,986 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264462986"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264462986"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264462986"}]},"ts":"1676264462986"} 2023-02-13 05:01:02,988 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=222, ppid=217, state=RUNNABLE; CloseRegionProcedure ded68bdbede054e91359e396257503f8, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:01:02,988 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=223, ppid=216, state=RUNNABLE; CloseRegionProcedure 6b4351d6197a4df3ac583e92c336c18d, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:01:02,989 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=224, ppid=221, state=RUNNABLE; CloseRegionProcedure ea5c27dda854cd4930a8099c7b035386, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:01:02,990 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=225, ppid=218, state=RUNNABLE; CloseRegionProcedure c6ef1845e0de3d153503ed9c1b81ead2, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:01:02,991 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=226, ppid=219, state=RUNNABLE; CloseRegionProcedure 3a6771e102cb35cd838b546f1db7ee49, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:01:02,991 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=220, ppid=215, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, UNASSIGN 2023-02-13 05:01:02,992 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=f11a86da37311510c40c261c7bcedac9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:02,992 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264462992"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264462992"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264462992"}]},"ts":"1676264462992"} 2023-02-13 05:01:02,993 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=227, ppid=220, state=RUNNABLE; CloseRegionProcedure f11a86da37311510c40c261c7bcedac9, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:01:03,140 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ded68bdbede054e91359e396257503f8 2023-02-13 05:01:03,140 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ded68bdbede054e91359e396257503f8, disabling compactions & flushes 2023-02-13 05:01:03,140 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:03,140 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:03,140 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. after waiting 0 ms 2023-02-13 05:01:03,140 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:03,142 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:03,142 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:03,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ea5c27dda854cd4930a8099c7b035386, disabling compactions & flushes 2023-02-13 05:01:03,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 6b4351d6197a4df3ac583e92c336c18d, disabling compactions & flushes 2023-02-13 05:01:03,142 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:03,142 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:03,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:03,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:03,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. after waiting 0 ms 2023-02-13 05:01:03,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:03,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. after waiting 0 ms 2023-02-13 05:01:03,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:03,144 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 05:01:03,145 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:03,146 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ded68bdbede054e91359e396257503f8: Waiting for close lock at 1676264463140Running coprocessor pre-close hooks at 1676264463140Disabling compacts and flushes for region at 1676264463140Disabling writes for close at 1676264463140Writing region close event to WAL at 1676264463141 (+1 ms)Running coprocessor post-close hooks at 1676264463145 (+4 ms)Closed at 1676264463145 2023-02-13 05:01:03,146 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 05:01:03,146 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 05:01:03,147 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:03,148 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 6b4351d6197a4df3ac583e92c336c18d: Waiting for close lock at 1676264463142Running coprocessor pre-close hooks at 1676264463142Disabling compacts and flushes for region at 1676264463142Disabling writes for close at 1676264463142Writing region close event to WAL at 1676264463143 (+1 ms)Running coprocessor post-close hooks at 1676264463147 (+4 ms)Closed at 1676264463147 2023-02-13 05:01:03,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ded68bdbede054e91359e396257503f8 2023-02-13 05:01:03,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:03,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:03,148 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f11a86da37311510c40c261c7bcedac9, disabling compactions & flushes 2023-02-13 05:01:03,148 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ea5c27dda854cd4930a8099c7b035386: Waiting for close lock at 1676264463142Running coprocessor pre-close hooks at 1676264463142Disabling compacts and flushes for region at 1676264463142Disabling writes for close at 1676264463142Writing region close event to WAL at 1676264463143 (+1 ms)Running coprocessor post-close hooks at 1676264463148 (+5 ms)Closed at 1676264463148 2023-02-13 05:01:03,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:03,148 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:03,148 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. after waiting 0 ms 2023-02-13 05:01:03,149 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:03,149 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=217 updating hbase:meta row=ded68bdbede054e91359e396257503f8, regionState=CLOSED 2023-02-13 05:01:03,149 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264463149"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264463149"}]},"ts":"1676264463149"} 2023-02-13 05:01:03,149 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:03,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:03,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 3a6771e102cb35cd838b546f1db7ee49, disabling compactions & flushes 2023-02-13 05:01:03,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:03,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:03,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. after waiting 0 ms 2023-02-13 05:01:03,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:03,150 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=216 updating hbase:meta row=6b4351d6197a4df3ac583e92c336c18d, regionState=CLOSED 2023-02-13 05:01:03,150 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264463150"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264463150"}]},"ts":"1676264463150"} 2023-02-13 05:01:03,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:03,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:03,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c6ef1845e0de3d153503ed9c1b81ead2, disabling compactions & flushes 2023-02-13 05:01:03,150 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:03,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:03,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. after waiting 0 ms 2023-02-13 05:01:03,150 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:03,151 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=221 updating hbase:meta row=ea5c27dda854cd4930a8099c7b035386, regionState=CLOSED 2023-02-13 05:01:03,151 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264463151"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264463151"}]},"ts":"1676264463151"} 2023-02-13 05:01:03,152 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 05:01:03,152 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=222, resume processing ppid=217 2023-02-13 05:01:03,152 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=222, ppid=217, state=SUCCESS; CloseRegionProcedure ded68bdbede054e91359e396257503f8, server=jenkins-hbase4.apache.org,34363,1676264434089 in 162 msec 2023-02-13 05:01:03,153 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:03,153 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f11a86da37311510c40c261c7bcedac9: Waiting for close lock at 1676264463148Running coprocessor pre-close hooks at 1676264463148Disabling compacts and flushes for region at 1676264463148Disabling writes for close at 1676264463149 (+1 ms)Writing region close event to WAL at 1676264463149Running coprocessor post-close hooks at 1676264463153 (+4 ms)Closed at 1676264463153 2023-02-13 05:01:03,154 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=223, resume processing ppid=216 2023-02-13 05:01:03,154 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=223, ppid=216, state=SUCCESS; CloseRegionProcedure 6b4351d6197a4df3ac583e92c336c18d, server=jenkins-hbase4.apache.org,39885,1676264434059 in 163 msec 2023-02-13 05:01:03,155 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=217, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, UNASSIGN in 168 msec 2023-02-13 05:01:03,155 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:03,155 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=224, resume processing ppid=221 2023-02-13 05:01:03,155 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=220 updating hbase:meta row=f11a86da37311510c40c261c7bcedac9, regionState=CLOSED 2023-02-13 05:01:03,156 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=216, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, UNASSIGN in 170 msec 2023-02-13 05:01:03,156 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264463155"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264463155"}]},"ts":"1676264463155"} 2023-02-13 05:01:03,156 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=224, ppid=221, state=SUCCESS; CloseRegionProcedure ea5c27dda854cd4930a8099c7b035386, server=jenkins-hbase4.apache.org,42589,1676264434015 in 164 msec 2023-02-13 05:01:03,156 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 05:01:03,159 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:03,159 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 3a6771e102cb35cd838b546f1db7ee49: Waiting for close lock at 1676264463150Running coprocessor pre-close hooks at 1676264463150Disabling compacts and flushes for region at 1676264463150Disabling writes for close at 1676264463150Writing region close event to WAL at 1676264463150Running coprocessor post-close hooks at 1676264463157 (+7 ms)Closed at 1676264463159 (+2 ms) 2023-02-13 05:01:03,159 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=221, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, UNASSIGN in 172 msec 2023-02-13 05:01:03,160 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-02-13 05:01:03,161 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:03,161 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c6ef1845e0de3d153503ed9c1b81ead2: Waiting for close lock at 1676264463150Running coprocessor pre-close hooks at 1676264463150Disabling compacts and flushes for region at 1676264463150Disabling writes for close at 1676264463150Writing region close event to WAL at 1676264463151 (+1 ms)Running coprocessor post-close hooks at 1676264463160 (+9 ms)Closed at 1676264463161 (+1 ms) 2023-02-13 05:01:03,161 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:03,161 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=219 updating hbase:meta row=3a6771e102cb35cd838b546f1db7ee49, regionState=CLOSED 2023-02-13 05:01:03,161 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264463161"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264463161"}]},"ts":"1676264463161"} 2023-02-13 05:01:03,162 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=227, resume processing ppid=220 2023-02-13 05:01:03,162 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=227, ppid=220, state=SUCCESS; CloseRegionProcedure f11a86da37311510c40c261c7bcedac9, server=jenkins-hbase4.apache.org,34363,1676264434089 in 166 msec 2023-02-13 05:01:03,162 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:03,162 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=218 updating hbase:meta row=c6ef1845e0de3d153503ed9c1b81ead2, regionState=CLOSED 2023-02-13 05:01:03,162 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264463162"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264463162"}]},"ts":"1676264463162"} 2023-02-13 05:01:03,163 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=220, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, UNASSIGN in 178 msec 2023-02-13 05:01:03,164 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=226, resume processing ppid=219 2023-02-13 05:01:03,164 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=226, ppid=219, state=SUCCESS; CloseRegionProcedure 3a6771e102cb35cd838b546f1db7ee49, server=jenkins-hbase4.apache.org,39885,1676264434059 in 171 msec 2023-02-13 05:01:03,164 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=225, resume processing ppid=218 2023-02-13 05:01:03,164 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=225, ppid=218, state=SUCCESS; CloseRegionProcedure c6ef1845e0de3d153503ed9c1b81ead2, server=jenkins-hbase4.apache.org,42589,1676264434015 in 173 msec 2023-02-13 05:01:03,164 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=219, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, UNASSIGN in 180 msec 2023-02-13 05:01:03,165 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=218, resume processing ppid=215 2023-02-13 05:01:03,165 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=218, ppid=215, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, UNASSIGN in 180 msec 2023-02-13 05:01:03,166 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676264461448","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264463166"}]},"ts":"1676264463166"} 2023-02-13 05:01:03,167 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676264461448, state=DISABLED in hbase:meta 2023-02-13 05:01:03,168 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676264461448 to state=DISABLED 2023-02-13 05:01:03,169 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=215, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676264461448 in 193 msec 2023-02-13 05:01:03,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=215 2023-02-13 05:01:03,231 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676264461448, procId: 215 completed 2023-02-13 05:01:03,231 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=emptySnaptb-1676264461448 table=testCorruptedSnapshot-1676264461448 type=FLUSH ttl=0 } 2023-02-13 05:01:03,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676264463231 (current time:1676264463231). 2023-02-13 05:01:03,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 05:01:03,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] zookeeper.ReadOnlyZKClient(139): Connect 0x2a528ad5 to 127.0.0.1:63943 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 05:01:03,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34a8b6d6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 05:01:03,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:01:03,239 INFO [RS-EventLoopGroup-4-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44488, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:01:03,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a528ad5 to 127.0.0.1:63943 2023-02-13 05:01:03,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:01:03,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 05:01:03,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=emptySnaptb-1676264461448 table=testCorruptedSnapshot-1676264461448 type=FLUSH ttl=0 } 2023-02-13 05:01:03,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676264461448, type=EXCLUSIVE 2023-02-13 05:01:03,243 DEBUG [PEWorker-1] locking.LockProcedure(309): LOCKED pid=228, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676264461448, type=EXCLUSIVE 2023-02-13 05:01:03,244 INFO [PEWorker-1] procedure2.TimeoutExecutorThread(81): ADDED pid=228, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676264461448, type=EXCLUSIVE; timeout=600000, timestamp=1676265063244 2023-02-13 05:01:03,244 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot emptySnaptb-1676264461448 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676264461448 2023-02-13 05:01:03,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(664): Started snapshot: { ss=emptySnaptb-1676264461448 table=testCorruptedSnapshot-1676264461448 type=FLUSH ttl=0 } 2023-02-13 05:01:03,244 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=emptySnaptb-1676264461448 table=testCorruptedSnapshot-1676264461448 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 05:01:03,244 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 05:01:03,260 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676264461448 table=testCorruptedSnapshot-1676264461448 type=DISABLED ttl=0 } 2023-02-13 05:01:03,264 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:01:03,264 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:01:03,265 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:01:03,264 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:01:03,265 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:01:03,265 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:01:03,265 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:01:03,264 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:01:03,265 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:01:03,265 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:01:03,265 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:01:03,265 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:01:03,265 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:01:03,266 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:01:03,266 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:01:03,266 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:01:03,266 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:01:03,268 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:01:03,268 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:01:03,268 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:01:03,269 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(318): No files under family: cf 2023-02-13 05:01:03,315 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=emptySnaptb-1676264461448 table=testCorruptedSnapshot-1676264461448 type=DISABLED ttl=0 } as finished. 2023-02-13 05:01:03,316 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for emptySnaptb-1676264461448 2023-02-13 05:01:03,316 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/emptySnaptb-1676264461448 2023-02-13 05:01:03,336 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/emptySnaptb-1676264461448 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/emptySnaptb-1676264461448 2023-02-13 05:01:03,353 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot emptySnaptb-1676264461448 of table testCorruptedSnapshot-1676264461448 completed 2023-02-13 05:01:03,353 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/emptySnaptb-1676264461448 2023-02-13 05:01:03,353 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/emptySnaptb-1676264461448 2023-02-13 05:01:03,353 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot emptySnaptb-1676264461448 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676264461448 at 1676264463244Starting to write region info and WALs for regions for offline snapshot:{ ss=emptySnaptb-1676264461448 table=testCorruptedSnapshot-1676264461448 type=DISABLED ttl=0 } at 1676264463260 (+16 ms)Consolidate snapshot: emptySnaptb-1676264461448 at 1676264463316 (+56 ms)Loading Region manifests for emptySnaptb-1676264461448 at 1676264463316Writing data manifest for emptySnaptb-1676264461448 at 1676264463319 (+3 ms)Verifying snapshot: emptySnaptb-1676264461448 at 1676264463330 (+11 ms)Snapshot emptySnaptb-1676264461448 of table testCorruptedSnapshot-1676264461448 completed at 1676264463353 (+23 ms) 2023-02-13 05:01:03,355 DEBUG [PEWorker-2] locking.LockProcedure(242): UNLOCKED pid=228, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676264461448, type=EXCLUSIVE 2023-02-13 05:01:03,356 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=228, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676264461448, type=EXCLUSIVE in 113 msec 2023-02-13 05:01:03,494 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 05:01:03,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=emptySnaptb-1676264461448 table=testCorruptedSnapshot-1676264461448 type=FLUSH ttl=0 } is done 2023-02-13 05:01:03,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(401): Snapshot '{ ss=emptySnaptb-1676264461448 table=testCorruptedSnapshot-1676264461448 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 05:01:03,496 INFO [Listener at localhost/42165] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676264461448 2023-02-13 05:01:03,496 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676264461448 2023-02-13 05:01:03,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=229, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676264461448 2023-02-13 05:01:03,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-13 05:01:03,500 INFO [PEWorker-3] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676264461448 2023-02-13 05:01:03,500 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676264461448","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264463500"}]},"ts":"1676264463500"} 2023-02-13 05:01:03,501 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676264461448, state=ENABLING in hbase:meta 2023-02-13 05:01:03,504 DEBUG [PEWorker-3] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 05:01:03,504 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, ASSIGN}, {pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, ASSIGN}, {pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, ASSIGN}, {pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, ASSIGN}, {pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, ASSIGN}, {pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, ASSIGN}] 2023-02-13 05:01:03,507 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, ASSIGN 2023-02-13 05:01:03,507 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, ASSIGN 2023-02-13 05:01:03,507 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, ASSIGN 2023-02-13 05:01:03,507 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, ASSIGN 2023-02-13 05:01:03,508 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:03,508 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=233, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39885,1676264434059; forceNewPlan=false, retain=true 2023-02-13 05:01:03,508 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:03,508 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:03,508 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=232, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42589,1676264434015; forceNewPlan=false, retain=true 2023-02-13 05:01:03,508 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=231, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,34363,1676264434089; forceNewPlan=false, retain=true 2023-02-13 05:01:03,508 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:03,508 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, ASSIGN 2023-02-13 05:01:03,508 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=230, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39885,1676264434059; forceNewPlan=false, retain=true 2023-02-13 05:01:03,508 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, ASSIGN 2023-02-13 05:01:03,509 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:03,509 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:03,509 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=235, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42589,1676264434015; forceNewPlan=false, retain=true 2023-02-13 05:01:03,509 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=234, ppid=229, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,34363,1676264434089; forceNewPlan=false, retain=true 2023-02-13 05:01:03,658 INFO [jenkins-hbase4:45905] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 05:01:03,658 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=c6ef1845e0de3d153503ed9c1b81ead2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:03,658 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=ded68bdbede054e91359e396257503f8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:03,658 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=3a6771e102cb35cd838b546f1db7ee49, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:03,659 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264463658"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264463658"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264463658"}]},"ts":"1676264463658"} 2023-02-13 05:01:03,658 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=6b4351d6197a4df3ac583e92c336c18d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:03,658 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=ea5c27dda854cd4930a8099c7b035386, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:03,659 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264463658"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264463658"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264463658"}]},"ts":"1676264463658"} 2023-02-13 05:01:03,659 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264463658"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264463658"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264463658"}]},"ts":"1676264463658"} 2023-02-13 05:01:03,659 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264463658"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264463658"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264463658"}]},"ts":"1676264463658"} 2023-02-13 05:01:03,659 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264463658"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264463658"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264463658"}]},"ts":"1676264463658"} 2023-02-13 05:01:03,660 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=236, ppid=231, state=RUNNABLE; OpenRegionProcedure ded68bdbede054e91359e396257503f8, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:01:03,660 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=237, ppid=230, state=RUNNABLE; OpenRegionProcedure 6b4351d6197a4df3ac583e92c336c18d, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:01:03,661 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=238, ppid=233, state=RUNNABLE; OpenRegionProcedure 3a6771e102cb35cd838b546f1db7ee49, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:01:03,661 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=239, ppid=232, state=RUNNABLE; OpenRegionProcedure c6ef1845e0de3d153503ed9c1b81ead2, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:01:03,665 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=240, ppid=235, state=RUNNABLE; OpenRegionProcedure ea5c27dda854cd4930a8099c7b035386, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:01:03,667 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=f11a86da37311510c40c261c7bcedac9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:03,667 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264463667"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264463667"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264463667"}]},"ts":"1676264463667"} 2023-02-13 05:01:03,669 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=241, ppid=234, state=RUNNABLE; OpenRegionProcedure f11a86da37311510c40c261c7bcedac9, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:01:03,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-13 05:01:03,818 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:03,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 3a6771e102cb35cd838b546f1db7ee49, NAME => 'testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 05:01:03,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676264461448 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:03,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:03,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:03,818 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:03,819 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:03,819 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:03,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c6ef1845e0de3d153503ed9c1b81ead2, NAME => 'testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 05:01:03,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ded68bdbede054e91359e396257503f8, NAME => 'testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 05:01:03,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676264461448 c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:03,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676264461448 ded68bdbede054e91359e396257503f8 2023-02-13 05:01:03,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:03,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:03,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:03,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ded68bdbede054e91359e396257503f8 2023-02-13 05:01:03,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:03,819 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ded68bdbede054e91359e396257503f8 2023-02-13 05:01:03,819 INFO [StoreOpener-3a6771e102cb35cd838b546f1db7ee49-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:03,820 INFO [StoreOpener-c6ef1845e0de3d153503ed9c1b81ead2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:03,820 INFO [StoreOpener-ded68bdbede054e91359e396257503f8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ded68bdbede054e91359e396257503f8 2023-02-13 05:01:03,820 DEBUG [StoreOpener-3a6771e102cb35cd838b546f1db7ee49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/cf 2023-02-13 05:01:03,820 DEBUG [StoreOpener-3a6771e102cb35cd838b546f1db7ee49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/cf 2023-02-13 05:01:03,821 INFO [StoreOpener-3a6771e102cb35cd838b546f1db7ee49-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3a6771e102cb35cd838b546f1db7ee49 columnFamilyName cf 2023-02-13 05:01:03,821 DEBUG [StoreOpener-c6ef1845e0de3d153503ed9c1b81ead2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/cf 2023-02-13 05:01:03,821 DEBUG [StoreOpener-ded68bdbede054e91359e396257503f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/cf 2023-02-13 05:01:03,821 DEBUG [StoreOpener-c6ef1845e0de3d153503ed9c1b81ead2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/cf 2023-02-13 05:01:03,821 DEBUG [StoreOpener-ded68bdbede054e91359e396257503f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/cf 2023-02-13 05:01:03,821 INFO [StoreOpener-ded68bdbede054e91359e396257503f8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ded68bdbede054e91359e396257503f8 columnFamilyName cf 2023-02-13 05:01:03,821 INFO [StoreOpener-c6ef1845e0de3d153503ed9c1b81ead2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c6ef1845e0de3d153503ed9c1b81ead2 columnFamilyName cf 2023-02-13 05:01:03,821 INFO [StoreOpener-3a6771e102cb35cd838b546f1db7ee49-1] regionserver.HStore(310): Store=3a6771e102cb35cd838b546f1db7ee49/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:01:03,822 INFO [StoreOpener-ded68bdbede054e91359e396257503f8-1] regionserver.HStore(310): Store=ded68bdbede054e91359e396257503f8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:01:03,822 INFO [StoreOpener-c6ef1845e0de3d153503ed9c1b81ead2-1] regionserver.HStore(310): Store=c6ef1845e0de3d153503ed9c1b81ead2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:01:03,822 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:03,823 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8 2023-02-13 05:01:03,823 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:03,823 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:03,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8 2023-02-13 05:01:03,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:03,825 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:03,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ded68bdbede054e91359e396257503f8 2023-02-13 05:01:03,826 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 3a6771e102cb35cd838b546f1db7ee49; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70878265, jitterRate=0.05616845190525055}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:01:03,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 3a6771e102cb35cd838b546f1db7ee49: Running coprocessor pre-open hook at 1676264463818Writing region info on filesystem at 1676264463818Initializing all the Stores at 1676264463819 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264463819Cleaning up temporary data from old regions at 1676264463823 (+4 ms)Cleaning up detritus from prior splits at 1676264463824 (+1 ms)Running coprocessor post-open hooks at 1676264463826 (+2 ms)Region opened successfully at 1676264463826 2023-02-13 05:01:03,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:03,827 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49., pid=238, masterSystemTime=1676264463815 2023-02-13 05:01:03,827 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ded68bdbede054e91359e396257503f8; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61862963, jitterRate=-0.07817001640796661}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:01:03,827 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ded68bdbede054e91359e396257503f8: Running coprocessor pre-open hook at 1676264463819Writing region info on filesystem at 1676264463819Initializing all the Stores at 1676264463820 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264463820Cleaning up temporary data from old regions at 1676264463824 (+4 ms)Cleaning up detritus from prior splits at 1676264463824Running coprocessor post-open hooks at 1676264463827 (+3 ms)Region opened successfully at 1676264463827 2023-02-13 05:01:03,828 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8., pid=236, masterSystemTime=1676264463817 2023-02-13 05:01:03,828 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c6ef1845e0de3d153503ed9c1b81ead2; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71989975, jitterRate=0.07273422181606293}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:01:03,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c6ef1845e0de3d153503ed9c1b81ead2: Running coprocessor pre-open hook at 1676264463819Writing region info on filesystem at 1676264463819Initializing all the Stores at 1676264463820 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264463820Cleaning up temporary data from old regions at 1676264463824 (+4 ms)Cleaning up detritus from prior splits at 1676264463824Running coprocessor post-open hooks at 1676264463828 (+4 ms)Region opened successfully at 1676264463828 2023-02-13 05:01:03,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:03,828 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:03,828 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:03,828 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6b4351d6197a4df3ac583e92c336c18d, NAME => 'testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 05:01:03,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676264461448 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:03,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:03,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:03,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:03,829 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=233 updating hbase:meta row=3a6771e102cb35cd838b546f1db7ee49, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:03,829 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2., pid=239, masterSystemTime=1676264463817 2023-02-13 05:01:03,829 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264463829"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264463829"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264463829"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264463829"}]},"ts":"1676264463829"} 2023-02-13 05:01:03,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:03,830 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=231 updating hbase:meta row=ded68bdbede054e91359e396257503f8, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:03,830 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:03,830 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264463830"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264463830"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264463830"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264463830"}]},"ts":"1676264463830"} 2023-02-13 05:01:03,830 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:03,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f11a86da37311510c40c261c7bcedac9, NAME => 'testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 05:01:03,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676264461448 f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:03,830 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:03,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:03,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:03,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:03,831 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:03,831 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:03,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ea5c27dda854cd4930a8099c7b035386, NAME => 'testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 05:01:03,831 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=232 updating hbase:meta row=c6ef1845e0de3d153503ed9c1b81ead2, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:03,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676264461448 ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:03,831 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264463831"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264463831"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264463831"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264463831"}]},"ts":"1676264463831"} 2023-02-13 05:01:03,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:03,831 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:03,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:03,833 INFO [StoreOpener-ea5c27dda854cd4930a8099c7b035386-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:03,833 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=238, resume processing ppid=233 2023-02-13 05:01:03,833 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=238, ppid=233, state=SUCCESS; OpenRegionProcedure 3a6771e102cb35cd838b546f1db7ee49, server=jenkins-hbase4.apache.org,39885,1676264434059 in 170 msec 2023-02-13 05:01:03,833 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=236, resume processing ppid=231 2023-02-13 05:01:03,833 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=236, ppid=231, state=SUCCESS; OpenRegionProcedure ded68bdbede054e91359e396257503f8, server=jenkins-hbase4.apache.org,34363,1676264434089 in 171 msec 2023-02-13 05:01:03,834 DEBUG [StoreOpener-ea5c27dda854cd4930a8099c7b035386-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/cf 2023-02-13 05:01:03,834 DEBUG [StoreOpener-ea5c27dda854cd4930a8099c7b035386-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/cf 2023-02-13 05:01:03,834 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=233, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, ASSIGN in 329 msec 2023-02-13 05:01:03,834 INFO [StoreOpener-ea5c27dda854cd4930a8099c7b035386-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ea5c27dda854cd4930a8099c7b035386 columnFamilyName cf 2023-02-13 05:01:03,835 INFO [StoreOpener-6b4351d6197a4df3ac583e92c336c18d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:03,835 INFO [StoreOpener-f11a86da37311510c40c261c7bcedac9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:03,835 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=239, resume processing ppid=232 2023-02-13 05:01:03,835 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=239, ppid=232, state=SUCCESS; OpenRegionProcedure c6ef1845e0de3d153503ed9c1b81ead2, server=jenkins-hbase4.apache.org,42589,1676264434015 in 172 msec 2023-02-13 05:01:03,835 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=231, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, ASSIGN in 329 msec 2023-02-13 05:01:03,836 DEBUG [StoreOpener-6b4351d6197a4df3ac583e92c336c18d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/cf 2023-02-13 05:01:03,836 DEBUG [StoreOpener-6b4351d6197a4df3ac583e92c336c18d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/cf 2023-02-13 05:01:03,836 INFO [StoreOpener-ea5c27dda854cd4930a8099c7b035386-1] regionserver.HStore(310): Store=ea5c27dda854cd4930a8099c7b035386/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:01:03,836 INFO [StoreOpener-6b4351d6197a4df3ac583e92c336c18d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6b4351d6197a4df3ac583e92c336c18d columnFamilyName cf 2023-02-13 05:01:03,836 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=232, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, ASSIGN in 331 msec 2023-02-13 05:01:03,836 DEBUG [StoreOpener-f11a86da37311510c40c261c7bcedac9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/cf 2023-02-13 05:01:03,836 DEBUG [StoreOpener-f11a86da37311510c40c261c7bcedac9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/cf 2023-02-13 05:01:03,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:03,837 INFO [StoreOpener-f11a86da37311510c40c261c7bcedac9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f11a86da37311510c40c261c7bcedac9 columnFamilyName cf 2023-02-13 05:01:03,837 INFO [StoreOpener-6b4351d6197a4df3ac583e92c336c18d-1] regionserver.HStore(310): Store=6b4351d6197a4df3ac583e92c336c18d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:01:03,837 INFO [StoreOpener-f11a86da37311510c40c261c7bcedac9-1] regionserver.HStore(310): Store=f11a86da37311510c40c261c7bcedac9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:01:03,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:03,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:03,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:03,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:03,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:03,840 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:03,841 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ea5c27dda854cd4930a8099c7b035386; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62366628, jitterRate=-0.07066482305526733}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:01:03,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ea5c27dda854cd4930a8099c7b035386: Running coprocessor pre-open hook at 1676264463832Writing region info on filesystem at 1676264463832Initializing all the Stores at 1676264463832Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264463832Cleaning up temporary data from old regions at 1676264463838 (+6 ms)Cleaning up detritus from prior splits at 1676264463838Running coprocessor post-open hooks at 1676264463841 (+3 ms)Region opened successfully at 1676264463841 2023-02-13 05:01:03,841 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386., pid=240, masterSystemTime=1676264463817 2023-02-13 05:01:03,841 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:03,842 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:03,842 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f11a86da37311510c40c261c7bcedac9; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72892893, jitterRate=0.08618874847888947}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:01:03,842 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f11a86da37311510c40c261c7bcedac9: Running coprocessor pre-open hook at 1676264463831Writing region info on filesystem at 1676264463831Initializing all the Stores at 1676264463835 (+4 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264463835Cleaning up temporary data from old regions at 1676264463839 (+4 ms)Cleaning up detritus from prior splits at 1676264463840 (+1 ms)Running coprocessor post-open hooks at 1676264463842 (+2 ms)Region opened successfully at 1676264463842 2023-02-13 05:01:03,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:03,843 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:03,843 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 6b4351d6197a4df3ac583e92c336c18d; next sequenceid=5; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60228863, jitterRate=-0.10252000391483307}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:01:03,843 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 6b4351d6197a4df3ac583e92c336c18d: Running coprocessor pre-open hook at 1676264463829Writing region info on filesystem at 1676264463829Initializing all the Stores at 1676264463834 (+5 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264463834Cleaning up temporary data from old regions at 1676264463839 (+5 ms)Cleaning up detritus from prior splits at 1676264463839Running coprocessor post-open hooks at 1676264463843 (+4 ms)Region opened successfully at 1676264463843 2023-02-13 05:01:03,843 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=235 updating hbase:meta row=ea5c27dda854cd4930a8099c7b035386, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:03,843 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9., pid=241, masterSystemTime=1676264463817 2023-02-13 05:01:03,843 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264463843"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264463843"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264463843"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264463843"}]},"ts":"1676264463843"} 2023-02-13 05:01:03,843 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d., pid=237, masterSystemTime=1676264463815 2023-02-13 05:01:03,844 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:03,844 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:03,845 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=234 updating hbase:meta row=f11a86da37311510c40c261c7bcedac9, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:03,845 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264463845"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264463845"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264463845"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264463845"}]},"ts":"1676264463845"} 2023-02-13 05:01:03,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:03,845 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:03,845 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=230 updating hbase:meta row=6b4351d6197a4df3ac583e92c336c18d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:03,845 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264463845"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264463845"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264463845"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264463845"}]},"ts":"1676264463845"} 2023-02-13 05:01:03,847 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=240, resume processing ppid=235 2023-02-13 05:01:03,847 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=240, ppid=235, state=SUCCESS; OpenRegionProcedure ea5c27dda854cd4930a8099c7b035386, server=jenkins-hbase4.apache.org,42589,1676264434015 in 179 msec 2023-02-13 05:01:03,848 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=241, resume processing ppid=234 2023-02-13 05:01:03,848 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=241, ppid=234, state=SUCCESS; OpenRegionProcedure f11a86da37311510c40c261c7bcedac9, server=jenkins-hbase4.apache.org,34363,1676264434089 in 177 msec 2023-02-13 05:01:03,849 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=235, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, ASSIGN in 343 msec 2023-02-13 05:01:03,849 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=237, resume processing ppid=230 2023-02-13 05:01:03,849 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=237, ppid=230, state=SUCCESS; OpenRegionProcedure 6b4351d6197a4df3ac583e92c336c18d, server=jenkins-hbase4.apache.org,39885,1676264434059 in 187 msec 2023-02-13 05:01:03,850 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=234, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, ASSIGN in 344 msec 2023-02-13 05:01:03,851 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=230, resume processing ppid=229 2023-02-13 05:01:03,851 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=230, ppid=229, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, ASSIGN in 345 msec 2023-02-13 05:01:03,851 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676264461448","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264463851"}]},"ts":"1676264463851"} 2023-02-13 05:01:03,853 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676264461448, state=ENABLED in hbase:meta 2023-02-13 05:01:03,854 INFO [PEWorker-2] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676264461448' was successfully enabled. 2023-02-13 05:01:03,856 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=229, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676264461448 in 358 msec 2023-02-13 05:01:04,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=229 2023-02-13 05:01:04,251 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676264461448, procId: 229 completed 2023-02-13 05:01:04,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:01:04,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:01:04,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:01:04,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:01:04,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:01:04,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:01:04,265 DEBUG [Listener at localhost/42165] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676264461448 2023-02-13 05:01:04,265 DEBUG [Listener at localhost/42165] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:04,265 INFO [Listener at localhost/42165] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 05:01:04,276 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 05:01:04,277 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 05:01:04,278 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 05:01:04,279 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 05:01:04,280 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 05:01:04,282 INFO [Listener at localhost/42165] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676264461448 2023-02-13 05:01:04,282 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676264461448 2023-02-13 05:01:04,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=242, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676264461448 2023-02-13 05:01:04,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-13 05:01:04,287 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676264461448","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264464286"}]},"ts":"1676264464286"} 2023-02-13 05:01:04,288 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676264461448, state=DISABLING in hbase:meta 2023-02-13 05:01:04,290 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676264461448 to state=DISABLING 2023-02-13 05:01:04,290 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, UNASSIGN}, {pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, UNASSIGN}, {pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, UNASSIGN}, {pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, UNASSIGN}, {pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, UNASSIGN}, {pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, UNASSIGN}] 2023-02-13 05:01:04,291 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=245, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, UNASSIGN 2023-02-13 05:01:04,291 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=243, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, UNASSIGN 2023-02-13 05:01:04,291 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=244, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, UNASSIGN 2023-02-13 05:01:04,292 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=246, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, UNASSIGN 2023-02-13 05:01:04,292 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=6b4351d6197a4df3ac583e92c336c18d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:04,292 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=c6ef1845e0de3d153503ed9c1b81ead2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:04,292 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=3a6771e102cb35cd838b546f1db7ee49, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:04,292 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264464292"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264464292"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264464292"}]},"ts":"1676264464292"} 2023-02-13 05:01:04,292 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=ded68bdbede054e91359e396257503f8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:04,292 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264464292"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264464292"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264464292"}]},"ts":"1676264464292"} 2023-02-13 05:01:04,293 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264464292"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264464292"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264464292"}]},"ts":"1676264464292"} 2023-02-13 05:01:04,292 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=248, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, UNASSIGN 2023-02-13 05:01:04,292 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264464292"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264464292"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264464292"}]},"ts":"1676264464292"} 2023-02-13 05:01:04,294 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=ea5c27dda854cd4930a8099c7b035386, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:04,294 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264464294"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264464294"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264464294"}]},"ts":"1676264464294"} 2023-02-13 05:01:04,294 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=249, ppid=243, state=RUNNABLE; CloseRegionProcedure 6b4351d6197a4df3ac583e92c336c18d, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:01:04,295 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=250, ppid=246, state=RUNNABLE; CloseRegionProcedure 3a6771e102cb35cd838b546f1db7ee49, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:01:04,296 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=251, ppid=244, state=RUNNABLE; CloseRegionProcedure ded68bdbede054e91359e396257503f8, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:01:04,297 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=252, ppid=245, state=RUNNABLE; CloseRegionProcedure c6ef1845e0de3d153503ed9c1b81ead2, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:01:04,297 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=253, ppid=248, state=RUNNABLE; CloseRegionProcedure ea5c27dda854cd4930a8099c7b035386, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:01:04,298 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=247, ppid=242, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, UNASSIGN 2023-02-13 05:01:04,299 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=f11a86da37311510c40c261c7bcedac9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:04,299 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264464299"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264464299"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264464299"}]},"ts":"1676264464299"} 2023-02-13 05:01:04,300 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=254, ppid=247, state=RUNNABLE; CloseRegionProcedure f11a86da37311510c40c261c7bcedac9, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:01:04,447 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:04,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 6b4351d6197a4df3ac583e92c336c18d, disabling compactions & flushes 2023-02-13 05:01:04,447 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:04,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:04,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. after waiting 0 ms 2023-02-13 05:01:04,447 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:04,447 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 6b4351d6197a4df3ac583e92c336c18d 1/1 column families, dataSize=1.83 KB heapSize=4.19 KB 2023-02-13 05:01:04,449 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ded68bdbede054e91359e396257503f8 2023-02-13 05:01:04,449 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:04,449 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ded68bdbede054e91359e396257503f8, disabling compactions & flushes 2023-02-13 05:01:04,449 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ea5c27dda854cd4930a8099c7b035386, disabling compactions & flushes 2023-02-13 05:01:04,449 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:04,449 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:04,449 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:04,449 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:04,449 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. after waiting 0 ms 2023-02-13 05:01:04,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:04,449 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. after waiting 0 ms 2023-02-13 05:01:04,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:04,450 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing ded68bdbede054e91359e396257503f8 1/1 column families, dataSize=1.76 KB heapSize=4.05 KB 2023-02-13 05:01:04,450 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing ea5c27dda854cd4930a8099c7b035386 1/1 column families, dataSize=22.63 KB heapSize=48.91 KB 2023-02-13 05:01:04,482 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b20230213c30cbc08e63043fca92446eee8c7ca4e to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/c4ca4238a0b923820dcc509a6f75849b20230213c30cbc08e63043fca92446eee8c7ca4e 2023-02-13 05:01:04,485 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202302135713c1945093465e94004b2e52f119aa to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/d41d8cd98f00b204e9800998ecf8427e202302135713c1945093465e94004b2e52f119aa 2023-02-13 05:01:04,487 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d52023021314ca154df2f64d1fa8b89d8d95e3f38c to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/e4da3b7fbbce2345d7772b0674a318d52023021314ca154df2f64d1fa8b89d8d95e3f38c 2023-02-13 05:01:04,494 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/.tmp/cf/f43d7530820648cb854f1d7e7c6ee2f3 2023-02-13 05:01:04,498 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/.tmp/cf/f43d7530820648cb854f1d7e7c6ee2f3 as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/cf/f43d7530820648cb854f1d7e7c6ee2f3 2023-02-13 05:01:04,501 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/cf/f43d7530820648cb854f1d7e7c6ee2f3, entries=27, sequenceid=8, filesize=9.6 K 2023-02-13 05:01:04,501 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/.tmp/cf/a20aad9ac1e34317ad796a7b1dce4b50 2023-02-13 05:01:04,502 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for ded68bdbede054e91359e396257503f8 in 52ms, sequenceid=8, compaction requested=false 2023-02-13 05:01:04,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'testCorruptedSnapshot-1676264461448' 2023-02-13 05:01:04,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=22.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/.tmp/cf/704985202f6747a49988dad80dd7e0e8 2023-02-13 05:01:04,506 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 05:01:04,507 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:04,507 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ded68bdbede054e91359e396257503f8: Waiting for close lock at 1676264464449Running coprocessor pre-close hooks at 1676264464449Disabling compacts and flushes for region at 1676264464449Disabling writes for close at 1676264464450 (+1 ms)Obtaining lock to block concurrent updates at 1676264464450Preparing flush snapshotting stores in ded68bdbede054e91359e396257503f8 at 1676264464450Finished memstore snapshotting testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8., syncing WAL and waiting on mvcc, flushsize=dataSize=1807, getHeapSize=4128, getOffHeapSize=0, getCellsCount=27 at 1676264464450Flushing stores of testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. at 1676264464450Flushing ded68bdbede054e91359e396257503f8/cf: creating writer at 1676264464451 (+1 ms)Flushing mob file ded68bdbede054e91359e396257503f8/cf: appending metadata at 1676264464455 (+4 ms)Flushing mob file ded68bdbede054e91359e396257503f8/cf: closing flushed file at 1676264464455Flushing ded68bdbede054e91359e396257503f8/cf: appending metadata at 1676264464483 (+28 ms)Flushing ded68bdbede054e91359e396257503f8/cf: closing flushed file at 1676264464483Flushing ded68bdbede054e91359e396257503f8/cf: reopening flushed file at 1676264464498 (+15 ms)Finished flush of dataSize ~1.76 KB/1807, heapSize ~4.03 KB/4128, currentSize=0 B/0 for ded68bdbede054e91359e396257503f8 in 52ms, sequenceid=8, compaction requested=false at 1676264464502 (+4 ms)Writing region close event to WAL at 1676264464503 (+1 ms)Running coprocessor post-close hooks at 1676264464506 (+3 ms)Closed at 1676264464507 (+1 ms) 2023-02-13 05:01:04,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/.tmp/cf/704985202f6747a49988dad80dd7e0e8 as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/cf/704985202f6747a49988dad80dd7e0e8 2023-02-13 05:01:04,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ded68bdbede054e91359e396257503f8 2023-02-13 05:01:04,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:04,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f11a86da37311510c40c261c7bcedac9, disabling compactions & flushes 2023-02-13 05:01:04,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:04,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:04,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. after waiting 0 ms 2023-02-13 05:01:04,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:04,508 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=244 updating hbase:meta row=ded68bdbede054e91359e396257503f8, regionState=CLOSED 2023-02-13 05:01:04,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing f11a86da37311510c40c261c7bcedac9 1/1 column families, dataSize=1.57 KB heapSize=3.63 KB 2023-02-13 05:01:04,508 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264464508"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264464508"}]},"ts":"1676264464508"} 2023-02-13 05:01:04,508 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/.tmp/cf/a20aad9ac1e34317ad796a7b1dce4b50 as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/cf/a20aad9ac1e34317ad796a7b1dce4b50 2023-02-13 05:01:04,513 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/cf/704985202f6747a49988dad80dd7e0e8, entries=346, sequenceid=8, filesize=64.5 K 2023-02-13 05:01:04,513 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=251, resume processing ppid=244 2023-02-13 05:01:04,513 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=251, ppid=244, state=SUCCESS; CloseRegionProcedure ded68bdbede054e91359e396257503f8, server=jenkins-hbase4.apache.org,34363,1676264434089 in 214 msec 2023-02-13 05:01:04,513 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/cf/a20aad9ac1e34317ad796a7b1dce4b50, entries=28, sequenceid=8, filesize=9.7 K 2023-02-13 05:01:04,513 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~22.63 KB/23178, heapSize ~48.89 KB/50064, currentSize=0 B/0 for ea5c27dda854cd4930a8099c7b035386 in 63ms, sequenceid=8, compaction requested=false 2023-02-13 05:01:04,515 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for 6b4351d6197a4df3ac583e92c336c18d in 67ms, sequenceid=8, compaction requested=false 2023-02-13 05:01:04,515 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=244, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, UNASSIGN in 223 msec 2023-02-13 05:01:04,517 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 05:01:04,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 05:01:04,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:04,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:04,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ea5c27dda854cd4930a8099c7b035386: Waiting for close lock at 1676264464449Running coprocessor pre-close hooks at 1676264464449Disabling compacts and flushes for region at 1676264464449Disabling writes for close at 1676264464450 (+1 ms)Obtaining lock to block concurrent updates at 1676264464450Preparing flush snapshotting stores in ea5c27dda854cd4930a8099c7b035386 at 1676264464450Finished memstore snapshotting testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386., syncing WAL and waiting on mvcc, flushsize=dataSize=23178, getHeapSize=50064, getOffHeapSize=0, getCellsCount=346 at 1676264464450Flushing stores of testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. at 1676264464450Flushing ea5c27dda854cd4930a8099c7b035386/cf: creating writer at 1676264464451 (+1 ms)Flushing mob file ea5c27dda854cd4930a8099c7b035386/cf: appending metadata at 1676264464458 (+7 ms)Flushing mob file ea5c27dda854cd4930a8099c7b035386/cf: closing flushed file at 1676264464458Flushing ea5c27dda854cd4930a8099c7b035386/cf: appending metadata at 1676264464488 (+30 ms)Flushing ea5c27dda854cd4930a8099c7b035386/cf: closing flushed file at 1676264464488Flushing ea5c27dda854cd4930a8099c7b035386/cf: reopening flushed file at 1676264464509 (+21 ms)Finished flush of dataSize ~22.63 KB/23178, heapSize ~48.89 KB/50064, currentSize=0 B/0 for ea5c27dda854cd4930a8099c7b035386 in 63ms, sequenceid=8, compaction requested=false at 1676264464513 (+4 ms)Writing region close event to WAL at 1676264464515 (+2 ms)Running coprocessor post-close hooks at 1676264464518 (+3 ms)Closed at 1676264464518 2023-02-13 05:01:04,518 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 6b4351d6197a4df3ac583e92c336c18d: Waiting for close lock at 1676264464447Running coprocessor pre-close hooks at 1676264464447Disabling compacts and flushes for region at 1676264464447Disabling writes for close at 1676264464447Obtaining lock to block concurrent updates at 1676264464447Preparing flush snapshotting stores in 6b4351d6197a4df3ac583e92c336c18d at 1676264464447Finished memstore snapshotting testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d., syncing WAL and waiting on mvcc, flushsize=dataSize=1874, getHeapSize=4272, getOffHeapSize=0, getCellsCount=28 at 1676264464447Flushing stores of testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. at 1676264464448 (+1 ms)Flushing 6b4351d6197a4df3ac583e92c336c18d/cf: creating writer at 1676264464449 (+1 ms)Flushing mob file 6b4351d6197a4df3ac583e92c336c18d/cf: appending metadata at 1676264464454 (+5 ms)Flushing mob file 6b4351d6197a4df3ac583e92c336c18d/cf: closing flushed file at 1676264464454Flushing 6b4351d6197a4df3ac583e92c336c18d/cf: appending metadata at 1676264464486 (+32 ms)Flushing 6b4351d6197a4df3ac583e92c336c18d/cf: closing flushed file at 1676264464486Flushing 6b4351d6197a4df3ac583e92c336c18d/cf: reopening flushed file at 1676264464509 (+23 ms)Finished flush of dataSize ~1.83 KB/1874, heapSize ~4.17 KB/4272, currentSize=0 B/0 for 6b4351d6197a4df3ac583e92c336c18d in 67ms, sequenceid=8, compaction requested=false at 1676264464515 (+6 ms)Writing region close event to WAL at 1676264464515Running coprocessor post-close hooks at 1676264464518 (+3 ms)Closed at 1676264464518 2023-02-13 05:01:04,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:04,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:04,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 3a6771e102cb35cd838b546f1db7ee49, disabling compactions & flushes 2023-02-13 05:01:04,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:04,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:04,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. after waiting 0 ms 2023-02-13 05:01:04,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:04,519 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 3a6771e102cb35cd838b546f1db7ee49 1/1 column families, dataSize=2.81 KB heapSize=6.30 KB 2023-02-13 05:01:04,520 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=243 updating hbase:meta row=6b4351d6197a4df3ac583e92c336c18d, regionState=CLOSED 2023-02-13 05:01:04,520 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264464520"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264464520"}]},"ts":"1676264464520"} 2023-02-13 05:01:04,520 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:04,520 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:04,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c6ef1845e0de3d153503ed9c1b81ead2, disabling compactions & flushes 2023-02-13 05:01:04,520 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:04,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:04,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. after waiting 0 ms 2023-02-13 05:01:04,520 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:04,520 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing c6ef1845e0de3d153503ed9c1b81ead2 1/1 column families, dataSize=2.09 KB heapSize=4.75 KB 2023-02-13 05:01:04,524 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=248 updating hbase:meta row=ea5c27dda854cd4930a8099c7b035386, regionState=CLOSED 2023-02-13 05:01:04,524 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264464522"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264464522"}]},"ts":"1676264464522"} 2023-02-13 05:01:04,527 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=249, resume processing ppid=243 2023-02-13 05:01:04,527 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=249, ppid=243, state=SUCCESS; CloseRegionProcedure 6b4351d6197a4df3ac583e92c336c18d, server=jenkins-hbase4.apache.org,39885,1676264434059 in 231 msec 2023-02-13 05:01:04,528 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=253, resume processing ppid=248 2023-02-13 05:01:04,528 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=253, ppid=248, state=SUCCESS; CloseRegionProcedure ea5c27dda854cd4930a8099c7b035386, server=jenkins-hbase4.apache.org,42589,1676264434015 in 229 msec 2023-02-13 05:01:04,529 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=243, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, UNASSIGN in 237 msec 2023-02-13 05:01:04,529 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=248, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, UNASSIGN in 238 msec 2023-02-13 05:01:04,530 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c20230213fd44fad68f464125acd60b506ff6b12b to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/a87ff679a2f3e71d9181a67b7542122c20230213fd44fad68f464125acd60b506ff6b12b 2023-02-13 05:01:04,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-13 05:01:04,547 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=1.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/.tmp/cf/94e86fe7895240e9950f90ec9b3fc26b 2023-02-13 05:01:04,548 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf320230213c6986118be3c4fed80e28eba52681475 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213c6986118be3c4fed80e28eba52681475 2023-02-13 05:01:04,550 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c202302134155ae02f4584596ba6e34d042b5500a to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/c81e728d9d4c2f636f067f89cc14862c202302134155ae02f4584596ba6e34d042b5500a 2023-02-13 05:01:04,551 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/.tmp/cf/94e86fe7895240e9950f90ec9b3fc26b as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/cf/94e86fe7895240e9950f90ec9b3fc26b 2023-02-13 05:01:04,554 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/cf/94e86fe7895240e9950f90ec9b3fc26b, entries=24, sequenceid=8, filesize=9.0 K 2023-02-13 05:01:04,555 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.57 KB/1606, heapSize ~3.61 KB/3696, currentSize=0 B/0 for f11a86da37311510c40c261c7bcedac9 in 47ms, sequenceid=8, compaction requested=false 2023-02-13 05:01:04,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 05:01:04,559 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/.tmp/cf/17388fd8e87944a4a8d08c6f08a3bb35 2023-02-13 05:01:04,559 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:04,559 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f11a86da37311510c40c261c7bcedac9: Waiting for close lock at 1676264464508Running coprocessor pre-close hooks at 1676264464508Disabling compacts and flushes for region at 1676264464508Disabling writes for close at 1676264464508Obtaining lock to block concurrent updates at 1676264464508Preparing flush snapshotting stores in f11a86da37311510c40c261c7bcedac9 at 1676264464508Finished memstore snapshotting testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9., syncing WAL and waiting on mvcc, flushsize=dataSize=1606, getHeapSize=3696, getOffHeapSize=0, getCellsCount=24 at 1676264464509 (+1 ms)Flushing stores of testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. at 1676264464509Flushing f11a86da37311510c40c261c7bcedac9/cf: creating writer at 1676264464510 (+1 ms)Flushing mob file f11a86da37311510c40c261c7bcedac9/cf: appending metadata at 1676264464515 (+5 ms)Flushing mob file f11a86da37311510c40c261c7bcedac9/cf: closing flushed file at 1676264464515Flushing f11a86da37311510c40c261c7bcedac9/cf: appending metadata at 1676264464530 (+15 ms)Flushing f11a86da37311510c40c261c7bcedac9/cf: closing flushed file at 1676264464531 (+1 ms)Flushing f11a86da37311510c40c261c7bcedac9/cf: reopening flushed file at 1676264464551 (+20 ms)Finished flush of dataSize ~1.57 KB/1606, heapSize ~3.61 KB/3696, currentSize=0 B/0 for f11a86da37311510c40c261c7bcedac9 in 47ms, sequenceid=8, compaction requested=false at 1676264464555 (+4 ms)Writing region close event to WAL at 1676264464556 (+1 ms)Running coprocessor post-close hooks at 1676264464559 (+3 ms)Closed at 1676264464559 2023-02-13 05:01:04,560 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/.tmp/cf/9492210216e1415bafcf4380ca4856c1 2023-02-13 05:01:04,561 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:04,561 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=247 updating hbase:meta row=f11a86da37311510c40c261c7bcedac9, regionState=CLOSED 2023-02-13 05:01:04,561 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264464561"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264464561"}]},"ts":"1676264464561"} 2023-02-13 05:01:04,563 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/.tmp/cf/17388fd8e87944a4a8d08c6f08a3bb35 as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/cf/17388fd8e87944a4a8d08c6f08a3bb35 2023-02-13 05:01:04,563 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/.tmp/cf/9492210216e1415bafcf4380ca4856c1 as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/cf/9492210216e1415bafcf4380ca4856c1 2023-02-13 05:01:04,566 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/cf/17388fd8e87944a4a8d08c6f08a3bb35, entries=43, sequenceid=8, filesize=12.3 K 2023-02-13 05:01:04,566 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=254, resume processing ppid=247 2023-02-13 05:01:04,566 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/cf/9492210216e1415bafcf4380ca4856c1, entries=32, sequenceid=8, filesize=10.4 K 2023-02-13 05:01:04,567 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=254, ppid=247, state=SUCCESS; CloseRegionProcedure f11a86da37311510c40c261c7bcedac9, server=jenkins-hbase4.apache.org,34363,1676264434089 in 263 msec 2023-02-13 05:01:04,567 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.81 KB/2879, heapSize ~6.28 KB/6432, currentSize=0 B/0 for 3a6771e102cb35cd838b546f1db7ee49 in 48ms, sequenceid=8, compaction requested=false 2023-02-13 05:01:04,567 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for c6ef1845e0de3d153503ed9c1b81ead2 in 47ms, sequenceid=8, compaction requested=false 2023-02-13 05:01:04,568 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=247, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, UNASSIGN in 276 msec 2023-02-13 05:01:04,571 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 05:01:04,571 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:04,571 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c6ef1845e0de3d153503ed9c1b81ead2: Waiting for close lock at 1676264464520Running coprocessor pre-close hooks at 1676264464520Disabling compacts and flushes for region at 1676264464520Disabling writes for close at 1676264464520Obtaining lock to block concurrent updates at 1676264464520Preparing flush snapshotting stores in c6ef1845e0de3d153503ed9c1b81ead2 at 1676264464520Finished memstore snapshotting testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2., syncing WAL and waiting on mvcc, flushsize=dataSize=2142, getHeapSize=4848, getOffHeapSize=0, getCellsCount=32 at 1676264464522 (+2 ms)Flushing stores of testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. at 1676264464522Flushing c6ef1845e0de3d153503ed9c1b81ead2/cf: creating writer at 1676264464524 (+2 ms)Flushing mob file c6ef1845e0de3d153503ed9c1b81ead2/cf: appending metadata at 1676264464530 (+6 ms)Flushing mob file c6ef1845e0de3d153503ed9c1b81ead2/cf: closing flushed file at 1676264464530Flushing c6ef1845e0de3d153503ed9c1b81ead2/cf: appending metadata at 1676264464551 (+21 ms)Flushing c6ef1845e0de3d153503ed9c1b81ead2/cf: closing flushed file at 1676264464551Flushing c6ef1845e0de3d153503ed9c1b81ead2/cf: reopening flushed file at 1676264464564 (+13 ms)Finished flush of dataSize ~2.09 KB/2142, heapSize ~4.73 KB/4848, currentSize=0 B/0 for c6ef1845e0de3d153503ed9c1b81ead2 in 47ms, sequenceid=8, compaction requested=false at 1676264464567 (+3 ms)Writing region close event to WAL at 1676264464568 (+1 ms)Running coprocessor post-close hooks at 1676264464571 (+3 ms)Closed at 1676264464571 2023-02-13 05:01:04,573 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:04,573 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=245 updating hbase:meta row=c6ef1845e0de3d153503ed9c1b81ead2, regionState=CLOSED 2023-02-13 05:01:04,573 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/recovered.edits/11.seqid, newMaxSeqId=11, maxSeqId=4 2023-02-13 05:01:04,573 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264464573"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264464573"}]},"ts":"1676264464573"} 2023-02-13 05:01:04,573 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:04,573 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 3a6771e102cb35cd838b546f1db7ee49: Waiting for close lock at 1676264464519Running coprocessor pre-close hooks at 1676264464519Disabling compacts and flushes for region at 1676264464519Disabling writes for close at 1676264464519Obtaining lock to block concurrent updates at 1676264464520 (+1 ms)Preparing flush snapshotting stores in 3a6771e102cb35cd838b546f1db7ee49 at 1676264464520Finished memstore snapshotting testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49., syncing WAL and waiting on mvcc, flushsize=dataSize=2879, getHeapSize=6432, getOffHeapSize=0, getCellsCount=43 at 1676264464520Flushing stores of testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. at 1676264464521 (+1 ms)Flushing 3a6771e102cb35cd838b546f1db7ee49/cf: creating writer at 1676264464521Flushing mob file 3a6771e102cb35cd838b546f1db7ee49/cf: appending metadata at 1676264464528 (+7 ms)Flushing mob file 3a6771e102cb35cd838b546f1db7ee49/cf: closing flushed file at 1676264464528Flushing 3a6771e102cb35cd838b546f1db7ee49/cf: appending metadata at 1676264464549 (+21 ms)Flushing 3a6771e102cb35cd838b546f1db7ee49/cf: closing flushed file at 1676264464549Flushing 3a6771e102cb35cd838b546f1db7ee49/cf: reopening flushed file at 1676264464563 (+14 ms)Finished flush of dataSize ~2.81 KB/2879, heapSize ~6.28 KB/6432, currentSize=0 B/0 for 3a6771e102cb35cd838b546f1db7ee49 in 48ms, sequenceid=8, compaction requested=false at 1676264464567 (+4 ms)Writing region close event to WAL at 1676264464568 (+1 ms)Running coprocessor post-close hooks at 1676264464573 (+5 ms)Closed at 1676264464573 2023-02-13 05:01:04,575 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:04,575 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=246 updating hbase:meta row=3a6771e102cb35cd838b546f1db7ee49, regionState=CLOSED 2023-02-13 05:01:04,575 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264464575"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264464575"}]},"ts":"1676264464575"} 2023-02-13 05:01:04,576 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=252, resume processing ppid=245 2023-02-13 05:01:04,576 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=252, ppid=245, state=SUCCESS; CloseRegionProcedure c6ef1845e0de3d153503ed9c1b81ead2, server=jenkins-hbase4.apache.org,42589,1676264434015 in 277 msec 2023-02-13 05:01:04,581 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=245, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, UNASSIGN in 286 msec 2023-02-13 05:01:04,581 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=250, resume processing ppid=246 2023-02-13 05:01:04,581 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=250, ppid=246, state=SUCCESS; CloseRegionProcedure 3a6771e102cb35cd838b546f1db7ee49, server=jenkins-hbase4.apache.org,39885,1676264434059 in 282 msec 2023-02-13 05:01:04,582 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=246, resume processing ppid=242 2023-02-13 05:01:04,583 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=246, ppid=242, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, UNASSIGN in 291 msec 2023-02-13 05:01:04,583 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676264461448","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264464583"}]},"ts":"1676264464583"} 2023-02-13 05:01:04,584 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676264461448, state=DISABLED in hbase:meta 2023-02-13 05:01:04,586 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676264461448 to state=DISABLED 2023-02-13 05:01:04,588 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=242, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676264461448 in 304 msec 2023-02-13 05:01:05,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=242 2023-02-13 05:01:05,038 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676264461448, procId: 242 completed 2023-02-13 05:01:05,039 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1580): Client=jenkins//172.31.14.131 snapshot request for:{ ss=snaptb0-1676264461448 table=testCorruptedSnapshot-1676264461448 type=FLUSH ttl=0 } 2023-02-13 05:01:05,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotDescriptionUtils(316): Creation time not specified, setting to:1676264465039 (current time:1676264465039). 2023-02-13 05:01:05,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotDescriptionUtils(332): Snapshot current TTL value: 0 resetting it to default value: 0 2023-02-13 05:01:05,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] zookeeper.ReadOnlyZKClient(139): Connect 0x4da144a3 to 127.0.0.1:63943 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-02-13 05:01:05,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@731b6be0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-02-13 05:01:05,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-02-13 05:01:05,050 INFO [RS-EventLoopGroup-4-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39692, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-02-13 05:01:05,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4da144a3 to 127.0.0.1:63943 2023-02-13 05:01:05,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:01:05,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(601): No existing snapshot, attempting snapshot... 2023-02-13 05:01:05,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(659): Table is disabled, running snapshot entirely on master for { ss=snaptb0-1676264461448 table=testCorruptedSnapshot-1676264461448 type=FLUSH ttl=0 } 2023-02-13 05:01:05,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676264461448, type=EXCLUSIVE 2023-02-13 05:01:05,054 DEBUG [PEWorker-5] locking.LockProcedure(309): LOCKED pid=255, state=RUNNABLE; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676264461448, type=EXCLUSIVE 2023-02-13 05:01:05,055 INFO [PEWorker-5] procedure2.TimeoutExecutorThread(81): ADDED pid=255, state=WAITING_TIMEOUT, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676264461448, type=EXCLUSIVE; timeout=600000, timestamp=1676265065055 2023-02-13 05:01:05,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(664): Started snapshot: { ss=snaptb0-1676264461448 table=testCorruptedSnapshot-1676264461448 type=FLUSH ttl=0 } 2023-02-13 05:01:05,055 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(174): Running DISABLED table snapshot snaptb0-1676264461448 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676264461448 2023-02-13 05:01:05,055 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2418): Waiting a max of 300000 ms for snapshot '{ ss=snaptb0-1676264461448 table=testCorruptedSnapshot-1676264461448 type=FLUSH ttl=0 }'' to complete. (max 50000 ms per retry) 2023-02-13 05:01:05,055 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2428): (#1) Sleeping: 250ms while waiting for snapshot completion. 2023-02-13 05:01:05,063 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(97): Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676264461448 table=testCorruptedSnapshot-1676264461448 type=DISABLED ttl=0 } 2023-02-13 05:01:05,068 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:01:05,069 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:01:05,069 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:01:05,069 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:01:05,069 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:01:05,069 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:01:05,069 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:01:05,068 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:01:05,069 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:01:05,069 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:01:05,069 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:01:05,069 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(297): Storing region-info for snapshot. 2023-02-13 05:01:05,069 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:01:05,069 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(302): Creating references for hfiles 2023-02-13 05:01:05,071 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/cf/94e86fe7895240e9950f90ec9b3fc26b] hfiles 2023-02-13 05:01:05,071 DEBUG [DisabledTableSnapshot-pool-0] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/cf/94e86fe7895240e9950f90ec9b3fc26b 2023-02-13 05:01:05,071 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/cf/a20aad9ac1e34317ad796a7b1dce4b50] hfiles 2023-02-13 05:01:05,071 DEBUG [DisabledTableSnapshot-pool-5] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/cf/a20aad9ac1e34317ad796a7b1dce4b50 2023-02-13 05:01:05,071 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/cf/9492210216e1415bafcf4380ca4856c1] hfiles 2023-02-13 05:01:05,071 DEBUG [DisabledTableSnapshot-pool-4] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/cf/9492210216e1415bafcf4380ca4856c1 2023-02-13 05:01:05,071 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/cf/f43d7530820648cb854f1d7e7c6ee2f3] hfiles 2023-02-13 05:01:05,071 DEBUG [DisabledTableSnapshot-pool-3] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/cf/f43d7530820648cb854f1d7e7c6ee2f3 2023-02-13 05:01:05,072 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/cf/704985202f6747a49988dad80dd7e0e8] hfiles 2023-02-13 05:01:05,072 DEBUG [DisabledTableSnapshot-pool-1] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/cf/704985202f6747a49988dad80dd7e0e8 2023-02-13 05:01:05,073 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/cf/17388fd8e87944a4a8d08c6f08a3bb35] hfiles 2023-02-13 05:01:05,073 DEBUG [DisabledTableSnapshot-pool-6] snapshot.SnapshotManifest(363): Adding reference for hfile (1/1): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/cf/17388fd8e87944a4a8d08c6f08a3bb35 2023-02-13 05:01:05,077 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(355): Adding snapshot references for [hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/a87ff679a2f3e71d9181a67b7542122c20230213fd44fad68f464125acd60b506ff6b12b, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/c4ca4238a0b923820dcc509a6f75849b20230213c30cbc08e63043fca92446eee8c7ca4e, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/c81e728d9d4c2f636f067f89cc14862c202302134155ae02f4584596ba6e34d042b5500a, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/d41d8cd98f00b204e9800998ecf8427e202302135713c1945093465e94004b2e52f119aa, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/e4da3b7fbbce2345d7772b0674a318d52023021314ca154df2f64d1fa8b89d8d95e3f38c, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213c6986118be3c4fed80e28eba52681475] hfiles 2023-02-13 05:01:05,077 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (1/6): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/a87ff679a2f3e71d9181a67b7542122c20230213fd44fad68f464125acd60b506ff6b12b 2023-02-13 05:01:05,077 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (2/6): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/c4ca4238a0b923820dcc509a6f75849b20230213c30cbc08e63043fca92446eee8c7ca4e 2023-02-13 05:01:05,077 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (3/6): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/c81e728d9d4c2f636f067f89cc14862c202302134155ae02f4584596ba6e34d042b5500a 2023-02-13 05:01:05,077 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (4/6): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/d41d8cd98f00b204e9800998ecf8427e202302135713c1945093465e94004b2e52f119aa 2023-02-13 05:01:05,077 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (5/6): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/e4da3b7fbbce2345d7772b0674a318d52023021314ca154df2f64d1fa8b89d8d95e3f38c 2023-02-13 05:01:05,077 DEBUG [DisabledTableSnapshot-pool-2] snapshot.SnapshotManifest(363): Adding reference for hfile (6/6): hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213c6986118be3c4fed80e28eba52681475 2023-02-13 05:01:05,096 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.DisabledTableSnapshotHandler(120): Marking snapshot{ ss=snaptb0-1676264461448 table=testCorruptedSnapshot-1676264461448 type=DISABLED ttl=0 } as finished. 2023-02-13 05:01:05,097 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifest(484): Convert to Single Snapshot Manifest for snaptb0-1676264461448 2023-02-13 05:01:05,097 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotManifestV1(126): No regions under directory:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/snaptb0-1676264461448 2023-02-13 05:01:05,114 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.SnapshotDescriptionUtils(404): Sentinel is done, just moving the snapshot from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/snaptb0-1676264461448 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/snaptb0-1676264461448 2023-02-13 05:01:05,129 INFO [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(229): Snapshot snaptb0-1676264461448 of table testCorruptedSnapshot-1676264461448 completed 2023-02-13 05:01:05,129 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(246): Launching cleanup of working dir:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/snaptb0-1676264461448 2023-02-13 05:01:05,130 ERROR [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(251): Couldn't delete snapshot working directory:hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.hbase-snapshot/.tmp/snaptb0-1676264461448 2023-02-13 05:01:05,130 DEBUG [MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0-0] snapshot.TakeSnapshotHandler(257): Table snapshot journal : Running DISABLED table snapshot snaptb0-1676264461448 C_M_SNAPSHOT_TABLE on table testCorruptedSnapshot-1676264461448 at 1676264465055Starting to write region info and WALs for regions for offline snapshot:{ ss=snaptb0-1676264461448 table=testCorruptedSnapshot-1676264461448 type=DISABLED ttl=0 } at 1676264465063 (+8 ms)Consolidate snapshot: snaptb0-1676264461448 at 1676264465096 (+33 ms)Loading Region manifests for snaptb0-1676264461448 at 1676264465097 (+1 ms)Writing data manifest for snaptb0-1676264461448 at 1676264465099 (+2 ms)Verifying snapshot: snaptb0-1676264461448 at 1676264465107 (+8 ms)Snapshot snaptb0-1676264461448 of table testCorruptedSnapshot-1676264461448 completed at 1676264465129 (+22 ms) 2023-02-13 05:01:05,131 DEBUG [PEWorker-4] locking.LockProcedure(242): UNLOCKED pid=255, state=RUNNABLE, locked=true; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676264461448, type=EXCLUSIVE 2023-02-13 05:01:05,132 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=255, state=SUCCESS; org.apache.hadoop.hbase.master.locking.LockProcedure, tableName=testCorruptedSnapshot-1676264461448, type=EXCLUSIVE in 79 msec 2023-02-13 05:01:05,305 DEBUG [Listener at localhost/42165] client.HBaseAdmin(2434): Getting current status of snapshot from master... 2023-02-13 05:01:05,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1209): Checking to see if snapshot from request:{ ss=snaptb0-1676264461448 table=testCorruptedSnapshot-1676264461448 type=FLUSH ttl=0 } is done 2023-02-13 05:01:05,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(401): Snapshot '{ ss=snaptb0-1676264461448 table=testCorruptedSnapshot-1676264461448 type=FLUSH ttl=0 }' has completed, notifying client. 2023-02-13 05:01:05,307 INFO [Listener at localhost/42165] client.HBaseAdmin$14(830): Started enable of testCorruptedSnapshot-1676264461448 2023-02-13 05:01:05,307 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$10(2372): Client=jenkins//172.31.14.131 enable testCorruptedSnapshot-1676264461448 2023-02-13 05:01:05,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=256, state=RUNNABLE:ENABLE_TABLE_PREPARE; EnableTableProcedure table=testCorruptedSnapshot-1676264461448 2023-02-13 05:01:05,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-13 05:01:05,311 INFO [PEWorker-2] procedure.EnableTableProcedure(296): Attempting to enable the table testCorruptedSnapshot-1676264461448 2023-02-13 05:01:05,311 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676264461448","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264465311"}]},"ts":"1676264465311"} 2023-02-13 05:01:05,312 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676264461448, state=ENABLING in hbase:meta 2023-02-13 05:01:05,315 DEBUG [PEWorker-2] procedure.EnableTableProcedure(107): No change in number of region replicas (configuredReplicaCount=1); assigning. 2023-02-13 05:01:05,315 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, ASSIGN}, {pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, ASSIGN}, {pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, ASSIGN}, {pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, ASSIGN}, {pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, ASSIGN}, {pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, ASSIGN}] 2023-02-13 05:01:05,316 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, ASSIGN 2023-02-13 05:01:05,316 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, ASSIGN 2023-02-13 05:01:05,316 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, ASSIGN 2023-02-13 05:01:05,316 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, ASSIGN 2023-02-13 05:01:05,316 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:05,317 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=258, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,34363,1676264434089; forceNewPlan=false, retain=true 2023-02-13 05:01:05,317 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:05,317 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:05,317 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=257, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39885,1676264434059; forceNewPlan=false, retain=true 2023-02-13 05:01:05,317 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=260, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,39885,1676264434059; forceNewPlan=false, retain=true 2023-02-13 05:01:05,317 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:05,317 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, ASSIGN 2023-02-13 05:01:05,318 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=259, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42589,1676264434015; forceNewPlan=false, retain=true 2023-02-13 05:01:05,318 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, ASSIGN 2023-02-13 05:01:05,318 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:05,318 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=261, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,34363,1676264434089; forceNewPlan=false, retain=true 2023-02-13 05:01:05,319 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(189): Setting lastHost as the region location jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:05,319 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=262, ppid=256, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, ASSIGN; state=CLOSED, location=jenkins-hbase4.apache.org,42589,1676264434015; forceNewPlan=false, retain=true 2023-02-13 05:01:05,467 INFO [jenkins-hbase4:45905] balancer.BaseLoadBalancer(1545): Reassigned 6 regions. 6 retained the pre-restart assignment. 2023-02-13 05:01:05,467 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=3a6771e102cb35cd838b546f1db7ee49, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:05,467 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=c6ef1845e0de3d153503ed9c1b81ead2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:05,467 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=ded68bdbede054e91359e396257503f8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:05,467 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=6b4351d6197a4df3ac583e92c336c18d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:05,467 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=ea5c27dda854cd4930a8099c7b035386, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:05,468 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264465467"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264465467"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264465467"}]},"ts":"1676264465467"} 2023-02-13 05:01:05,467 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264465467"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264465467"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264465467"}]},"ts":"1676264465467"} 2023-02-13 05:01:05,467 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264465467"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264465467"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264465467"}]},"ts":"1676264465467"} 2023-02-13 05:01:05,467 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264465467"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264465467"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264465467"}]},"ts":"1676264465467"} 2023-02-13 05:01:05,468 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264465467"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264465467"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264465467"}]},"ts":"1676264465467"} 2023-02-13 05:01:05,469 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=263, ppid=257, state=RUNNABLE; OpenRegionProcedure 6b4351d6197a4df3ac583e92c336c18d, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:01:05,470 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=264, ppid=258, state=RUNNABLE; OpenRegionProcedure ded68bdbede054e91359e396257503f8, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:01:05,470 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=265, ppid=259, state=RUNNABLE; OpenRegionProcedure c6ef1845e0de3d153503ed9c1b81ead2, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:01:05,471 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=266, ppid=260, state=RUNNABLE; OpenRegionProcedure 3a6771e102cb35cd838b546f1db7ee49, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:01:05,471 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=267, ppid=262, state=RUNNABLE; OpenRegionProcedure ea5c27dda854cd4930a8099c7b035386, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:01:05,472 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=f11a86da37311510c40c261c7bcedac9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:05,472 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264465472"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264465472"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264465472"}]},"ts":"1676264465472"} 2023-02-13 05:01:05,473 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=268, ppid=261, state=RUNNABLE; OpenRegionProcedure f11a86da37311510c40c261c7bcedac9, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:01:05,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-13 05:01:05,623 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:05,623 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:05,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 3a6771e102cb35cd838b546f1db7ee49, NAME => 'testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.', STARTKEY => '3', ENDKEY => '4'} 2023-02-13 05:01:05,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ded68bdbede054e91359e396257503f8, NAME => 'testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.', STARTKEY => '1', ENDKEY => '2'} 2023-02-13 05:01:05,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676264461448 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:05,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:05,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676264461448 ded68bdbede054e91359e396257503f8 2023-02-13 05:01:05,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:05,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:05,623 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:05,624 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ded68bdbede054e91359e396257503f8 2023-02-13 05:01:05,623 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:05,624 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => ea5c27dda854cd4930a8099c7b035386, NAME => 'testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.', STARTKEY => '5', ENDKEY => ''} 2023-02-13 05:01:05,624 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ded68bdbede054e91359e396257503f8 2023-02-13 05:01:05,624 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676264461448 ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:05,624 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:05,624 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:05,624 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:05,625 INFO [StoreOpener-3a6771e102cb35cd838b546f1db7ee49-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:05,625 INFO [StoreOpener-ded68bdbede054e91359e396257503f8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ded68bdbede054e91359e396257503f8 2023-02-13 05:01:05,625 INFO [StoreOpener-ea5c27dda854cd4930a8099c7b035386-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:05,625 DEBUG [StoreOpener-3a6771e102cb35cd838b546f1db7ee49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/cf 2023-02-13 05:01:05,625 DEBUG [StoreOpener-3a6771e102cb35cd838b546f1db7ee49-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/cf 2023-02-13 05:01:05,625 DEBUG [StoreOpener-ded68bdbede054e91359e396257503f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/cf 2023-02-13 05:01:05,626 DEBUG [StoreOpener-ded68bdbede054e91359e396257503f8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/cf 2023-02-13 05:01:05,626 DEBUG [StoreOpener-ea5c27dda854cd4930a8099c7b035386-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/cf 2023-02-13 05:01:05,626 DEBUG [StoreOpener-ea5c27dda854cd4930a8099c7b035386-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/cf 2023-02-13 05:01:05,626 INFO [StoreOpener-3a6771e102cb35cd838b546f1db7ee49-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3a6771e102cb35cd838b546f1db7ee49 columnFamilyName cf 2023-02-13 05:01:05,626 INFO [StoreOpener-ea5c27dda854cd4930a8099c7b035386-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ea5c27dda854cd4930a8099c7b035386 columnFamilyName cf 2023-02-13 05:01:05,626 INFO [StoreOpener-ded68bdbede054e91359e396257503f8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ded68bdbede054e91359e396257503f8 columnFamilyName cf 2023-02-13 05:01:05,630 DEBUG [StoreOpener-ea5c27dda854cd4930a8099c7b035386-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/cf/704985202f6747a49988dad80dd7e0e8 2023-02-13 05:01:05,630 DEBUG [StoreOpener-ded68bdbede054e91359e396257503f8-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/cf/f43d7530820648cb854f1d7e7c6ee2f3 2023-02-13 05:01:05,630 DEBUG [StoreOpener-3a6771e102cb35cd838b546f1db7ee49-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/cf/17388fd8e87944a4a8d08c6f08a3bb35 2023-02-13 05:01:05,630 INFO [StoreOpener-ea5c27dda854cd4930a8099c7b035386-1] regionserver.HStore(310): Store=ea5c27dda854cd4930a8099c7b035386/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:01:05,630 INFO [StoreOpener-ded68bdbede054e91359e396257503f8-1] regionserver.HStore(310): Store=ded68bdbede054e91359e396257503f8/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:01:05,630 INFO [StoreOpener-3a6771e102cb35cd838b546f1db7ee49-1] regionserver.HStore(310): Store=3a6771e102cb35cd838b546f1db7ee49/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:01:05,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:05,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:05,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8 2023-02-13 05:01:05,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:05,631 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:05,632 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8 2023-02-13 05:01:05,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:05,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for ded68bdbede054e91359e396257503f8 2023-02-13 05:01:05,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:05,634 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ea5c27dda854cd4930a8099c7b035386; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74767586, jitterRate=0.11412385106086731}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:01:05,634 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 3a6771e102cb35cd838b546f1db7ee49; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66900679, jitterRate=-0.0031021982431411743}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:01:05,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ea5c27dda854cd4930a8099c7b035386: Running coprocessor pre-open hook at 1676264465624Writing region info on filesystem at 1676264465624Initializing all the Stores at 1676264465624Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264465625 (+1 ms)Cleaning up temporary data from old regions at 1676264465632 (+7 ms)Cleaning up detritus from prior splits at 1676264465632Running coprocessor post-open hooks at 1676264465634 (+2 ms)Region opened successfully at 1676264465634 2023-02-13 05:01:05,634 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened ded68bdbede054e91359e396257503f8; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68307379, jitterRate=0.017859265208244324}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:01:05,634 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 3a6771e102cb35cd838b546f1db7ee49: Running coprocessor pre-open hook at 1676264465624Writing region info on filesystem at 1676264465624Initializing all the Stores at 1676264465624Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264465624Cleaning up temporary data from old regions at 1676264465632 (+8 ms)Cleaning up detritus from prior splits at 1676264465632Running coprocessor post-open hooks at 1676264465634 (+2 ms)Region opened successfully at 1676264465634 2023-02-13 05:01:05,635 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for ded68bdbede054e91359e396257503f8: Running coprocessor pre-open hook at 1676264465624Writing region info on filesystem at 1676264465624Initializing all the Stores at 1676264465624Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264465624Cleaning up temporary data from old regions at 1676264465632 (+8 ms)Cleaning up detritus from prior splits at 1676264465632Running coprocessor post-open hooks at 1676264465635 (+3 ms)Region opened successfully at 1676264465635 2023-02-13 05:01:05,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49., pid=266, masterSystemTime=1676264465621 2023-02-13 05:01:05,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386., pid=267, masterSystemTime=1676264465622 2023-02-13 05:01:05,635 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8., pid=264, masterSystemTime=1676264465621 2023-02-13 05:01:05,636 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:05,636 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:05,636 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:05,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => 6b4351d6197a4df3ac583e92c336c18d, NAME => 'testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.', STARTKEY => '', ENDKEY => '1'} 2023-02-13 05:01:05,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676264461448 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:05,637 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=260 updating hbase:meta row=3a6771e102cb35cd838b546f1db7ee49, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:05,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:05,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:05,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:05,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:05,637 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264465637"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264465637"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264465637"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264465637"}]},"ts":"1676264465637"} 2023-02-13 05:01:05,637 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:05,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:05,637 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => c6ef1845e0de3d153503ed9c1b81ead2, NAME => 'testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.', STARTKEY => '2', ENDKEY => '3'} 2023-02-13 05:01:05,637 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=262 updating hbase:meta row=ea5c27dda854cd4930a8099c7b035386, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:05,637 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264465637"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264465637"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264465637"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264465637"}]},"ts":"1676264465637"} 2023-02-13 05:01:05,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:05,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676264461448 c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:05,638 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:05,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:05,638 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:05,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:05,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:05,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7850): Opening region: {ENCODED => f11a86da37311510c40c261c7bcedac9, NAME => 'testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.', STARTKEY => '4', ENDKEY => '5'} 2023-02-13 05:01:05,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table testCorruptedSnapshot-1676264461448 f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:05,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(865): Instantiated testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-02-13 05:01:05,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7890): checking encryption for f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:05,638 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7893): checking classloading for f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:05,639 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=258 updating hbase:meta row=ded68bdbede054e91359e396257503f8, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:05,639 INFO [StoreOpener-6b4351d6197a4df3ac583e92c336c18d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:05,639 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264465639"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264465639"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264465639"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264465639"}]},"ts":"1676264465639"} 2023-02-13 05:01:05,639 INFO [StoreOpener-c6ef1845e0de3d153503ed9c1b81ead2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:05,640 INFO [StoreOpener-f11a86da37311510c40c261c7bcedac9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family cf of region f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:05,640 DEBUG [StoreOpener-c6ef1845e0de3d153503ed9c1b81ead2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/cf 2023-02-13 05:01:05,640 DEBUG [StoreOpener-c6ef1845e0de3d153503ed9c1b81ead2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/cf 2023-02-13 05:01:05,640 INFO [StoreOpener-c6ef1845e0de3d153503ed9c1b81ead2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c6ef1845e0de3d153503ed9c1b81ead2 columnFamilyName cf 2023-02-13 05:01:05,641 DEBUG [StoreOpener-6b4351d6197a4df3ac583e92c336c18d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/cf 2023-02-13 05:01:05,641 DEBUG [StoreOpener-6b4351d6197a4df3ac583e92c336c18d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/cf 2023-02-13 05:01:05,641 INFO [StoreOpener-6b4351d6197a4df3ac583e92c336c18d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6b4351d6197a4df3ac583e92c336c18d columnFamilyName cf 2023-02-13 05:01:05,644 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=266, resume processing ppid=260 2023-02-13 05:01:05,644 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=266, ppid=260, state=SUCCESS; OpenRegionProcedure 3a6771e102cb35cd838b546f1db7ee49, server=jenkins-hbase4.apache.org,39885,1676264434059 in 168 msec 2023-02-13 05:01:05,645 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=267, resume processing ppid=262 2023-02-13 05:01:05,645 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=267, ppid=262, state=SUCCESS; OpenRegionProcedure ea5c27dda854cd4930a8099c7b035386, server=jenkins-hbase4.apache.org,42589,1676264434015 in 168 msec 2023-02-13 05:01:05,645 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=260, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, ASSIGN in 329 msec 2023-02-13 05:01:05,646 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=264, resume processing ppid=258 2023-02-13 05:01:05,646 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=264, ppid=258, state=SUCCESS; OpenRegionProcedure ded68bdbede054e91359e396257503f8, server=jenkins-hbase4.apache.org,34363,1676264434089 in 170 msec 2023-02-13 05:01:05,646 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=262, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, ASSIGN in 330 msec 2023-02-13 05:01:05,646 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=258, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, ASSIGN in 331 msec 2023-02-13 05:01:05,647 DEBUG [StoreOpener-f11a86da37311510c40c261c7bcedac9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/cf 2023-02-13 05:01:05,647 DEBUG [StoreOpener-f11a86da37311510c40c261c7bcedac9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/cf 2023-02-13 05:01:05,648 INFO [StoreOpener-f11a86da37311510c40c261c7bcedac9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:10, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f11a86da37311510c40c261c7bcedac9 columnFamilyName cf 2023-02-13 05:01:05,651 DEBUG [StoreOpener-c6ef1845e0de3d153503ed9c1b81ead2-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/cf/9492210216e1415bafcf4380ca4856c1 2023-02-13 05:01:05,651 INFO [StoreOpener-c6ef1845e0de3d153503ed9c1b81ead2-1] regionserver.HStore(310): Store=c6ef1845e0de3d153503ed9c1b81ead2/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:01:05,651 DEBUG [StoreOpener-6b4351d6197a4df3ac583e92c336c18d-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/cf/a20aad9ac1e34317ad796a7b1dce4b50 2023-02-13 05:01:05,651 INFO [StoreOpener-6b4351d6197a4df3ac583e92c336c18d-1] regionserver.HStore(310): Store=6b4351d6197a4df3ac583e92c336c18d/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:01:05,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:05,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:05,652 DEBUG [StoreOpener-f11a86da37311510c40c261c7bcedac9-1] regionserver.HStore(539): loaded hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/cf/94e86fe7895240e9950f90ec9b3fc26b 2023-02-13 05:01:05,652 INFO [StoreOpener-f11a86da37311510c40c261c7bcedac9-1] regionserver.HStore(310): Store=f11a86da37311510c40c261c7bcedac9/cf, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-02-13 05:01:05,652 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:05,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:05,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:05,653 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5208): Found 0 recovered edits file(s) under hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:05,655 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:05,655 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:05,656 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened 6b4351d6197a4df3ac583e92c336c18d; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74296258, jitterRate=0.10710051655769348}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:01:05,656 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for 6b4351d6197a4df3ac583e92c336c18d: Running coprocessor pre-open hook at 1676264465637Writing region info on filesystem at 1676264465637Initializing all the Stores at 1676264465638 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264465638Cleaning up temporary data from old regions at 1676264465653 (+15 ms)Cleaning up detritus from prior splits at 1676264465653Running coprocessor post-open hooks at 1676264465656 (+3 ms)Region opened successfully at 1676264465656 2023-02-13 05:01:05,656 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1054): writing seq id for f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:05,656 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened c6ef1845e0de3d153503ed9c1b81ead2; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61137264, jitterRate=-0.08898377418518066}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:01:05,656 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for c6ef1845e0de3d153503ed9c1b81ead2: Running coprocessor pre-open hook at 1676264465638Writing region info on filesystem at 1676264465638Initializing all the Stores at 1676264465639 (+1 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264465639Cleaning up temporary data from old regions at 1676264465653 (+14 ms)Cleaning up detritus from prior splits at 1676264465653Running coprocessor post-open hooks at 1676264465656 (+3 ms)Region opened successfully at 1676264465656 2023-02-13 05:01:05,656 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d., pid=263, masterSystemTime=1676264465621 2023-02-13 05:01:05,657 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2., pid=265, masterSystemTime=1676264465622 2023-02-13 05:01:05,657 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1071): Opened f11a86da37311510c40c261c7bcedac9; next sequenceid=12; SteppingSplitPolicysuper{IncreasingToUpperBoundRegionSplitPolicy{initialSize=268435456, ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67116573, jitterRate=1.1487305164337158E-4}}}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-02-13 05:01:05,657 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(964): Region open journal for f11a86da37311510c40c261c7bcedac9: Running coprocessor pre-open hook at 1676264465638Writing region info on filesystem at 1676264465638Initializing all the Stores at 1676264465640 (+2 ms)Instantiating store for column family {NAME => 'cf', MOB_THRESHOLD => '0', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} at 1676264465640Cleaning up temporary data from old regions at 1676264465654 (+14 ms)Cleaning up detritus from prior splits at 1676264465654Running coprocessor post-open hooks at 1676264465657 (+3 ms)Region opened successfully at 1676264465657 2023-02-13 05:01:05,657 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9., pid=268, masterSystemTime=1676264465621 2023-02-13 05:01:05,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:05,658 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:05,658 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=257 updating hbase:meta row=6b4351d6197a4df3ac583e92c336c18d, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:05,658 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:05,658 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:05,658 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264465658"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264465658"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264465658"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264465658"}]},"ts":"1676264465658"} 2023-02-13 05:01:05,659 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=259 updating hbase:meta row=c6ef1845e0de3d153503ed9c1b81ead2, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:05,659 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264465659"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264465659"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264465659"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264465659"}]},"ts":"1676264465659"} 2023-02-13 05:01:05,659 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:05,659 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:05,659 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=261 updating hbase:meta row=f11a86da37311510c40c261c7bcedac9, regionState=OPEN, openSeqNum=12, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:05,660 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264465659"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1676264465659"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1676264465659"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1676264465659"}]},"ts":"1676264465659"} 2023-02-13 05:01:05,661 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=263, resume processing ppid=257 2023-02-13 05:01:05,661 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=263, ppid=257, state=SUCCESS; OpenRegionProcedure 6b4351d6197a4df3ac583e92c336c18d, server=jenkins-hbase4.apache.org,39885,1676264434059 in 191 msec 2023-02-13 05:01:05,662 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=265, resume processing ppid=259 2023-02-13 05:01:05,662 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=265, ppid=259, state=SUCCESS; OpenRegionProcedure c6ef1845e0de3d153503ed9c1b81ead2, server=jenkins-hbase4.apache.org,42589,1676264434015 in 190 msec 2023-02-13 05:01:05,662 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=268, resume processing ppid=261 2023-02-13 05:01:05,663 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=257, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, ASSIGN in 346 msec 2023-02-13 05:01:05,663 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=268, ppid=261, state=SUCCESS; OpenRegionProcedure f11a86da37311510c40c261c7bcedac9, server=jenkins-hbase4.apache.org,34363,1676264434089 in 188 msec 2023-02-13 05:01:05,663 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=259, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, ASSIGN in 347 msec 2023-02-13 05:01:05,664 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=261, resume processing ppid=256 2023-02-13 05:01:05,664 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=261, ppid=256, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, ASSIGN in 348 msec 2023-02-13 05:01:05,664 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676264461448","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264465664"}]},"ts":"1676264465664"} 2023-02-13 05:01:05,665 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676264461448, state=ENABLED in hbase:meta 2023-02-13 05:01:05,667 INFO [PEWorker-5] procedure.EnableTableProcedure(310): Table 'testCorruptedSnapshot-1676264461448' was successfully enabled. 2023-02-13 05:01:05,668 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=256, state=SUCCESS; EnableTableProcedure table=testCorruptedSnapshot-1676264461448 in 360 msec 2023-02-13 05:01:06,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=256 2023-02-13 05:01:06,062 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: ENABLE, Table Name: default:testCorruptedSnapshot-1676264461448, procId: 256 completed 2023-02-13 05:01:06,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:01:06,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:01:06,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=34363] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:01:06,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:01:06,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=39885] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:01:06,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42589] regionserver.HRegion(8801): writing data to region testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. with WAL disabled. Data may be lost in the event of a crash. 2023-02-13 05:01:06,074 DEBUG [Listener at localhost/42165] hbase.HBaseTestingUtility(2627): Found 6 regions for table testCorruptedSnapshot-1676264461448 2023-02-13 05:01:06,074 DEBUG [Listener at localhost/42165] hbase.HBaseTestingUtility(2633): firstRegionName=testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:06,074 INFO [Listener at localhost/42165] hbase.Waiter(180): Waiting up to [60,000] milli-secs(wait.for.ratio=[1]) 2023-02-13 05:01:06,160 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '1', inclusive 2023-02-13 05:01:06,232 DEBUG [Finalizer] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6eba1460 to 127.0.0.1:63943 2023-02-13 05:01:06,232 DEBUG [Finalizer] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:01:06,252 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '2', inclusive 2023-02-13 05:01:06,323 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '3', inclusive 2023-02-13 05:01:06,420 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '4', inclusive 2023-02-13 05:01:06,475 DEBUG [Listener at localhost/42165] client.ClientScanner(240): Advancing internal scanner to startKey at '5', inclusive 2023-02-13 05:01:07,249 ERROR [VerifySnapshot-pool-0] snapshot.SnapshotReferenceUtil(301): Can't find hfile: a87ff679a2f3e71d9181a67b7542122c20230213fd44fad68f464125acd60b506ff6b12b in the real (hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/a87ff679a2f3e71d9181a67b7542122c20230213fd44fad68f464125acd60b506ff6b12b) or archive (hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/a87ff679a2f3e71d9181a67b7542122c20230213fd44fad68f464125acd60b506ff6b12b) directory for the primary table. 2023-02-13 05:01:07,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.MetricsHBaseServer(134): Unknown exception type org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: a87ff679a2f3e71d9181a67b7542122c20230213fd44fad68f464125acd60b506ff6b12b in the real (hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/a87ff679a2f3e71d9181a67b7542122c20230213fd44fad68f464125acd60b506ff6b12b) or archive (hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/a87ff679a2f3e71d9181a67b7542122c20230213fd44fad68f464125acd60b506ff6b12b) directory for the primary table. at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:225) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.concurrentVisitReferencedFiles(SnapshotReferenceUtil.java:187) at org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.verifySnapshot(SnapshotReferenceUtil.java:164) at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreOrCloneSnapshot(SnapshotManager.java:827) at org.apache.hadoop.hbase.master.HMaster$14.run(HMaster.java:2493) at org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:132) at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2489) at org.apache.hadoop.hbase.master.MasterRpcServices.restoreSnapshot(MasterRpcServices.java:1469) at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-02-13 05:01:07,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] ipc.CallRunner(144): callId: 362 service: MasterService methodName: RestoreSnapshot size: 115 connection: 172.31.14.131:59360 deadline: 1676264527245, exception=org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Can't find hfile: a87ff679a2f3e71d9181a67b7542122c20230213fd44fad68f464125acd60b506ff6b12b in the real (hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/a87ff679a2f3e71d9181a67b7542122c20230213fd44fad68f464125acd60b506ff6b12b) or archive (hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/a87ff679a2f3e71d9181a67b7542122c20230213fd44fad68f464125acd60b506ff6b12b) directory for the primary table. 2023-02-13 05:01:07,253 INFO [Listener at localhost/42165] client.HBaseAdmin$15(890): Started disable of testCorruptedSnapshot-1676264461448 2023-02-13 05:01:07,253 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$11(2405): Client=jenkins//172.31.14.131 disable testCorruptedSnapshot-1676264461448 2023-02-13 05:01:07,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=270, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=testCorruptedSnapshot-1676264461448 2023-02-13 05:01:07,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-13 05:01:07,256 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676264461448","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264467256"}]},"ts":"1676264467256"} 2023-02-13 05:01:07,256 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676264461448, state=DISABLING in hbase:meta 2023-02-13 05:01:07,258 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set testCorruptedSnapshot-1676264461448 to state=DISABLING 2023-02-13 05:01:07,259 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, UNASSIGN}, {pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, UNASSIGN}, {pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, UNASSIGN}, {pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, UNASSIGN}, {pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, UNASSIGN}, {pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, UNASSIGN}] 2023-02-13 05:01:07,259 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=273, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, UNASSIGN 2023-02-13 05:01:07,259 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=272, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, UNASSIGN 2023-02-13 05:01:07,259 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=271, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, UNASSIGN 2023-02-13 05:01:07,259 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=276, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, UNASSIGN 2023-02-13 05:01:07,260 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=275, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, UNASSIGN 2023-02-13 05:01:07,260 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=c6ef1845e0de3d153503ed9c1b81ead2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:07,260 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=ded68bdbede054e91359e396257503f8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:07,260 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=6b4351d6197a4df3ac583e92c336c18d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:07,260 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=ea5c27dda854cd4930a8099c7b035386, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:07,260 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264467260"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264467260"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264467260"}]},"ts":"1676264467260"} 2023-02-13 05:01:07,260 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264467260"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264467260"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264467260"}]},"ts":"1676264467260"} 2023-02-13 05:01:07,260 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=f11a86da37311510c40c261c7bcedac9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:07,260 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264467260"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264467260"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264467260"}]},"ts":"1676264467260"} 2023-02-13 05:01:07,261 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264467260"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264467260"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264467260"}]},"ts":"1676264467260"} 2023-02-13 05:01:07,260 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264467260"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264467260"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264467260"}]},"ts":"1676264467260"} 2023-02-13 05:01:07,261 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=277, ppid=271, state=RUNNABLE; CloseRegionProcedure 6b4351d6197a4df3ac583e92c336c18d, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:01:07,262 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=278, ppid=276, state=RUNNABLE; CloseRegionProcedure ea5c27dda854cd4930a8099c7b035386, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:01:07,262 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=279, ppid=272, state=RUNNABLE; CloseRegionProcedure ded68bdbede054e91359e396257503f8, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:01:07,263 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=280, ppid=275, state=RUNNABLE; CloseRegionProcedure f11a86da37311510c40c261c7bcedac9, server=jenkins-hbase4.apache.org,34363,1676264434089}] 2023-02-13 05:01:07,263 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=281, ppid=273, state=RUNNABLE; CloseRegionProcedure c6ef1845e0de3d153503ed9c1b81ead2, server=jenkins-hbase4.apache.org,42589,1676264434015}] 2023-02-13 05:01:07,264 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=274, ppid=270, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, UNASSIGN 2023-02-13 05:01:07,265 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=3a6771e102cb35cd838b546f1db7ee49, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:07,265 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264467265"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1676264467265"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1676264467265"}]},"ts":"1676264467265"} 2023-02-13 05:01:07,266 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=282, ppid=274, state=RUNNABLE; CloseRegionProcedure 3a6771e102cb35cd838b546f1db7ee49, server=jenkins-hbase4.apache.org,39885,1676264434059}] 2023-02-13 05:01:07,414 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:07,414 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:07,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing c6ef1845e0de3d153503ed9c1b81ead2, disabling compactions & flushes 2023-02-13 05:01:07,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 3a6771e102cb35cd838b546f1db7ee49, disabling compactions & flushes 2023-02-13 05:01:07,414 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:07,414 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:07,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:07,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:07,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. after waiting 0 ms 2023-02-13 05:01:07,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:07,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. after waiting 0 ms 2023-02-13 05:01:07,414 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:07,414 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing c6ef1845e0de3d153503ed9c1b81ead2 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-13 05:01:07,414 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 3a6771e102cb35cd838b546f1db7ee49 1/1 column families, dataSize=2.03 KB heapSize=4.61 KB 2023-02-13 05:01:07,415 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:07,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing f11a86da37311510c40c261c7bcedac9, disabling compactions & flushes 2023-02-13 05:01:07,415 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:07,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:07,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. after waiting 0 ms 2023-02-13 05:01:07,415 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:07,415 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing f11a86da37311510c40c261c7bcedac9 1/1 column families, dataSize=1.96 KB heapSize=4.47 KB 2023-02-13 05:01:07,437 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/a87ff679a2f3e71d9181a67b7542122c202302131d035aa4667f479ab53da427dc6719e7 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/a87ff679a2f3e71d9181a67b7542122c202302131d035aa4667f479ab53da427dc6719e7 2023-02-13 05:01:07,438 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132b2a0afef79c4e7bb4a3a765d0b33947 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132b2a0afef79c4e7bb4a3a765d0b33947 2023-02-13 05:01:07,438 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/c81e728d9d4c2f636f067f89cc14862c20230213e80a32a16b364b9a812c370d393a783e to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/c81e728d9d4c2f636f067f89cc14862c20230213e80a32a16b364b9a812c370d393a783e 2023-02-13 05:01:07,448 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/.tmp/cf/d1c935e662f84c06ad6d3e3688364edc 2023-02-13 05:01:07,448 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/.tmp/cf/7022918372ea489b9310c9c7dd7ebf70 2023-02-13 05:01:07,449 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/.tmp/cf/decb23750b4d445e8daed2f043c924b2 2023-02-13 05:01:07,451 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/.tmp/cf/d1c935e662f84c06ad6d3e3688364edc as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/cf/d1c935e662f84c06ad6d3e3688364edc 2023-02-13 05:01:07,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/.tmp/cf/7022918372ea489b9310c9c7dd7ebf70 as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/cf/7022918372ea489b9310c9c7dd7ebf70 2023-02-13 05:01:07,452 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/.tmp/cf/decb23750b4d445e8daed2f043c924b2 as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/cf/decb23750b4d445e8daed2f043c924b2 2023-02-13 05:01:07,455 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/cf/d1c935e662f84c06ad6d3e3688364edc, entries=30, sequenceid=15, filesize=10.1 K 2023-02-13 05:01:07,455 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/cf/7022918372ea489b9310c9c7dd7ebf70, entries=31, sequenceid=15, filesize=10.2 K 2023-02-13 05:01:07,455 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/cf/decb23750b4d445e8daed2f043c924b2, entries=31, sequenceid=15, filesize=10.2 K 2023-02-13 05:01:07,458 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for c6ef1845e0de3d153503ed9c1b81ead2 in 42ms, sequenceid=15, compaction requested=false 2023-02-13 05:01:07,458 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for f11a86da37311510c40c261c7bcedac9 in 43ms, sequenceid=15, compaction requested=false 2023-02-13 05:01:07,458 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 3a6771e102cb35cd838b546f1db7ee49 in 42ms, sequenceid=15, compaction requested=false 2023-02-13 05:01:07,461 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 05:01:07,461 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 05:01:07,461 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 05:01:07,462 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. 2023-02-13 05:01:07,462 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. 2023-02-13 05:01:07,462 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. 2023-02-13 05:01:07,462 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for f11a86da37311510c40c261c7bcedac9: Waiting for close lock at 1676264467415Running coprocessor pre-close hooks at 1676264467415Disabling compacts and flushes for region at 1676264467415Disabling writes for close at 1676264467415Obtaining lock to block concurrent updates at 1676264467416 (+1 ms)Preparing flush snapshotting stores in f11a86da37311510c40c261c7bcedac9 at 1676264467416Finished memstore snapshotting testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9., syncing WAL and waiting on mvcc, flushsize=dataSize=2008, getHeapSize=4560, getOffHeapSize=0, getCellsCount=30 at 1676264467416Flushing stores of testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9. at 1676264467417 (+1 ms)Flushing f11a86da37311510c40c261c7bcedac9/cf: creating writer at 1676264467417Flushing mob file f11a86da37311510c40c261c7bcedac9/cf: appending metadata at 1676264467421 (+4 ms)Flushing mob file f11a86da37311510c40c261c7bcedac9/cf: closing flushed file at 1676264467421Flushing f11a86da37311510c40c261c7bcedac9/cf: appending metadata at 1676264467438 (+17 ms)Flushing f11a86da37311510c40c261c7bcedac9/cf: closing flushed file at 1676264467438Flushing f11a86da37311510c40c261c7bcedac9/cf: reopening flushed file at 1676264467452 (+14 ms)Finished flush of dataSize ~1.96 KB/2008, heapSize ~4.45 KB/4560, currentSize=0 B/0 for f11a86da37311510c40c261c7bcedac9 in 43ms, sequenceid=15, compaction requested=false at 1676264467458 (+6 ms)Writing region close event to WAL at 1676264467458Running coprocessor post-close hooks at 1676264467461 (+3 ms)Closed at 1676264467462 (+1 ms) 2023-02-13 05:01:07,462 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 3a6771e102cb35cd838b546f1db7ee49: Waiting for close lock at 1676264467414Running coprocessor pre-close hooks at 1676264467414Disabling compacts and flushes for region at 1676264467414Disabling writes for close at 1676264467414Obtaining lock to block concurrent updates at 1676264467414Preparing flush snapshotting stores in 3a6771e102cb35cd838b546f1db7ee49 at 1676264467414Finished memstore snapshotting testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49., syncing WAL and waiting on mvcc, flushsize=dataSize=2075, getHeapSize=4704, getOffHeapSize=0, getCellsCount=31 at 1676264467415 (+1 ms)Flushing stores of testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49. at 1676264467415Flushing 3a6771e102cb35cd838b546f1db7ee49/cf: creating writer at 1676264467416 (+1 ms)Flushing mob file 3a6771e102cb35cd838b546f1db7ee49/cf: appending metadata at 1676264467421 (+5 ms)Flushing mob file 3a6771e102cb35cd838b546f1db7ee49/cf: closing flushed file at 1676264467421Flushing 3a6771e102cb35cd838b546f1db7ee49/cf: appending metadata at 1676264467439 (+18 ms)Flushing 3a6771e102cb35cd838b546f1db7ee49/cf: closing flushed file at 1676264467439Flushing 3a6771e102cb35cd838b546f1db7ee49/cf: reopening flushed file at 1676264467452 (+13 ms)Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for 3a6771e102cb35cd838b546f1db7ee49 in 42ms, sequenceid=15, compaction requested=false at 1676264467458 (+6 ms)Writing region close event to WAL at 1676264467459 (+1 ms)Running coprocessor post-close hooks at 1676264467462 (+3 ms)Closed at 1676264467462 2023-02-13 05:01:07,462 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for c6ef1845e0de3d153503ed9c1b81ead2: Waiting for close lock at 1676264467414Running coprocessor pre-close hooks at 1676264467414Disabling compacts and flushes for region at 1676264467414Disabling writes for close at 1676264467414Obtaining lock to block concurrent updates at 1676264467414Preparing flush snapshotting stores in c6ef1845e0de3d153503ed9c1b81ead2 at 1676264467414Finished memstore snapshotting testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2., syncing WAL and waiting on mvcc, flushsize=dataSize=2075, getHeapSize=4704, getOffHeapSize=0, getCellsCount=31 at 1676264467415 (+1 ms)Flushing stores of testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2. at 1676264467415Flushing c6ef1845e0de3d153503ed9c1b81ead2/cf: creating writer at 1676264467416 (+1 ms)Flushing mob file c6ef1845e0de3d153503ed9c1b81ead2/cf: appending metadata at 1676264467421 (+5 ms)Flushing mob file c6ef1845e0de3d153503ed9c1b81ead2/cf: closing flushed file at 1676264467421Flushing c6ef1845e0de3d153503ed9c1b81ead2/cf: appending metadata at 1676264467439 (+18 ms)Flushing c6ef1845e0de3d153503ed9c1b81ead2/cf: closing flushed file at 1676264467439Flushing c6ef1845e0de3d153503ed9c1b81ead2/cf: reopening flushed file at 1676264467452 (+13 ms)Finished flush of dataSize ~2.03 KB/2075, heapSize ~4.59 KB/4704, currentSize=0 B/0 for c6ef1845e0de3d153503ed9c1b81ead2 in 42ms, sequenceid=15, compaction requested=false at 1676264467458 (+6 ms)Writing region close event to WAL at 1676264467458Running coprocessor post-close hooks at 1676264467462 (+4 ms)Closed at 1676264467462 2023-02-13 05:01:07,463 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:07,463 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ded68bdbede054e91359e396257503f8 2023-02-13 05:01:07,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ded68bdbede054e91359e396257503f8, disabling compactions & flushes 2023-02-13 05:01:07,464 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:07,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:07,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. after waiting 0 ms 2023-02-13 05:01:07,464 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:07,464 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing ded68bdbede054e91359e396257503f8 1/1 column families, dataSize=2.55 KB heapSize=5.73 KB 2023-02-13 05:01:07,464 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=275 updating hbase:meta row=f11a86da37311510c40c261c7bcedac9, regionState=CLOSED 2023-02-13 05:01:07,464 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264467464"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264467464"}]},"ts":"1676264467464"} 2023-02-13 05:01:07,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:07,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:07,465 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing ea5c27dda854cd4930a8099c7b035386, disabling compactions & flushes 2023-02-13 05:01:07,465 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:07,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:07,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. after waiting 0 ms 2023-02-13 05:01:07,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:07,466 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=273 updating hbase:meta row=c6ef1845e0de3d153503ed9c1b81ead2, regionState=CLOSED 2023-02-13 05:01:07,466 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing ea5c27dda854cd4930a8099c7b035386 1/1 column families, dataSize=21.78 KB heapSize=47.08 KB 2023-02-13 05:01:07,466 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264467466"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264467466"}]},"ts":"1676264467466"} 2023-02-13 05:01:07,466 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:07,466 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:07,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 6b4351d6197a4df3ac583e92c336c18d, disabling compactions & flushes 2023-02-13 05:01:07,466 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:07,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:07,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. after waiting 0 ms 2023-02-13 05:01:07,466 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:07,466 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 6b4351d6197a4df3ac583e92c336c18d 1/1 column families, dataSize=2.35 KB heapSize=5.31 KB 2023-02-13 05:01:07,469 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=274 updating hbase:meta row=3a6771e102cb35cd838b546f1db7ee49, regionState=CLOSED 2023-02-13 05:01:07,469 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264467468"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264467468"}]},"ts":"1676264467468"} 2023-02-13 05:01:07,472 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=280, resume processing ppid=275 2023-02-13 05:01:07,472 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=280, ppid=275, state=SUCCESS; CloseRegionProcedure f11a86da37311510c40c261c7bcedac9, server=jenkins-hbase4.apache.org,34363,1676264434089 in 206 msec 2023-02-13 05:01:07,473 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=281, resume processing ppid=273 2023-02-13 05:01:07,473 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=281, ppid=273, state=SUCCESS; CloseRegionProcedure c6ef1845e0de3d153503ed9c1b81ead2, server=jenkins-hbase4.apache.org,42589,1676264434015 in 208 msec 2023-02-13 05:01:07,474 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=282, resume processing ppid=274 2023-02-13 05:01:07,474 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=275, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=f11a86da37311510c40c261c7bcedac9, UNASSIGN in 214 msec 2023-02-13 05:01:07,474 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=282, ppid=274, state=SUCCESS; CloseRegionProcedure 3a6771e102cb35cd838b546f1db7ee49, server=jenkins-hbase4.apache.org,39885,1676264434059 in 206 msec 2023-02-13 05:01:07,476 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=273, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=c6ef1845e0de3d153503ed9c1b81ead2, UNASSIGN in 215 msec 2023-02-13 05:01:07,476 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=274, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=3a6771e102cb35cd838b546f1db7ee49, UNASSIGN in 216 msec 2023-02-13 05:01:07,483 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/c4ca4238a0b923820dcc509a6f75849b202302134d8391bc734f4720a4702a1800a803a1 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/c4ca4238a0b923820dcc509a6f75849b202302134d8391bc734f4720a4702a1800a803a1 2023-02-13 05:01:07,486 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/e4da3b7fbbce2345d7772b0674a318d52023021394120bb78ef749e68429ae92143d3472 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/e4da3b7fbbce2345d7772b0674a318d52023021394120bb78ef749e68429ae92143d3472 2023-02-13 05:01:07,486 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230213a1ec7568501c4cc4b8fc1f8884db77cc to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/d41d8cd98f00b204e9800998ecf8427e20230213a1ec7568501c4cc4b8fc1f8884db77cc 2023-02-13 05:01:07,490 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/.tmp/cf/df5537851dc2498f9ae0922d6a77a7b1 2023-02-13 05:01:07,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/.tmp/cf/df5537851dc2498f9ae0922d6a77a7b1 as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/cf/df5537851dc2498f9ae0922d6a77a7b1 2023-02-13 05:01:07,496 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/cf/df5537851dc2498f9ae0922d6a77a7b1, entries=39, sequenceid=15, filesize=11.6 K 2023-02-13 05:01:07,496 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for ded68bdbede054e91359e396257503f8 in 32ms, sequenceid=15, compaction requested=false 2023-02-13 05:01:07,499 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=2.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/.tmp/cf/198c1b1681fe4d02899483305542ce83 2023-02-13 05:01:07,499 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=21.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/.tmp/cf/8599a2aec0ea4917aa50e068b59c2e13 2023-02-13 05:01:07,501 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 05:01:07,501 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. 2023-02-13 05:01:07,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ded68bdbede054e91359e396257503f8: Waiting for close lock at 1676264467464Running coprocessor pre-close hooks at 1676264467464Disabling compacts and flushes for region at 1676264467464Disabling writes for close at 1676264467464Obtaining lock to block concurrent updates at 1676264467464Preparing flush snapshotting stores in ded68bdbede054e91359e396257503f8 at 1676264467464Finished memstore snapshotting testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8., syncing WAL and waiting on mvcc, flushsize=dataSize=2611, getHeapSize=5856, getOffHeapSize=0, getCellsCount=39 at 1676264467464Flushing stores of testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8. at 1676264467465 (+1 ms)Flushing ded68bdbede054e91359e396257503f8/cf: creating writer at 1676264467466 (+1 ms)Flushing mob file ded68bdbede054e91359e396257503f8/cf: appending metadata at 1676264467472 (+6 ms)Flushing mob file ded68bdbede054e91359e396257503f8/cf: closing flushed file at 1676264467472Flushing ded68bdbede054e91359e396257503f8/cf: appending metadata at 1676264467484 (+12 ms)Flushing ded68bdbede054e91359e396257503f8/cf: closing flushed file at 1676264467484Flushing ded68bdbede054e91359e396257503f8/cf: reopening flushed file at 1676264467494 (+10 ms)Finished flush of dataSize ~2.55 KB/2611, heapSize ~5.72 KB/5856, currentSize=0 B/0 for ded68bdbede054e91359e396257503f8 in 32ms, sequenceid=15, compaction requested=false at 1676264467496 (+2 ms)Writing region close event to WAL at 1676264467498 (+2 ms)Running coprocessor post-close hooks at 1676264467501 (+3 ms)Closed at 1676264467501 2023-02-13 05:01:07,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/.tmp/cf/198c1b1681fe4d02899483305542ce83 as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/cf/198c1b1681fe4d02899483305542ce83 2023-02-13 05:01:07,502 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/.tmp/cf/8599a2aec0ea4917aa50e068b59c2e13 as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/cf/8599a2aec0ea4917aa50e068b59c2e13 2023-02-13 05:01:07,503 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ded68bdbede054e91359e396257503f8 2023-02-13 05:01:07,503 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=272 updating hbase:meta row=ded68bdbede054e91359e396257503f8, regionState=CLOSED 2023-02-13 05:01:07,503 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.","families":{"info":[{"qualifier":"regioninfo","vlen":71,"tag":[],"timestamp":"1676264467503"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264467503"}]},"ts":"1676264467503"} 2023-02-13 05:01:07,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/cf/198c1b1681fe4d02899483305542ce83, entries=36, sequenceid=15, filesize=11.1 K 2023-02-13 05:01:07,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/cf/8599a2aec0ea4917aa50e068b59c2e13, entries=333, sequenceid=15, filesize=62.3 K 2023-02-13 05:01:07,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 6b4351d6197a4df3ac583e92c336c18d in 40ms, sequenceid=15, compaction requested=false 2023-02-13 05:01:07,506 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=279, resume processing ppid=272 2023-02-13 05:01:07,506 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=279, ppid=272, state=SUCCESS; CloseRegionProcedure ded68bdbede054e91359e396257503f8, server=jenkins-hbase4.apache.org,34363,1676264434089 in 242 msec 2023-02-13 05:01:07,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-13 05:01:07,506 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~21.78 KB/22307, heapSize ~47.06 KB/48192, currentSize=0 B/0 for ea5c27dda854cd4930a8099c7b035386 in 40ms, sequenceid=15, compaction requested=false 2023-02-13 05:01:07,507 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=272, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ded68bdbede054e91359e396257503f8, UNASSIGN in 248 msec 2023-02-13 05:01:07,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 05:01:07,509 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/recovered.edits/18.seqid, newMaxSeqId=18, maxSeqId=11 2023-02-13 05:01:07,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. 2023-02-13 05:01:07,510 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. 2023-02-13 05:01:07,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 6b4351d6197a4df3ac583e92c336c18d: Waiting for close lock at 1676264467466Running coprocessor pre-close hooks at 1676264467466Disabling compacts and flushes for region at 1676264467466Disabling writes for close at 1676264467466Obtaining lock to block concurrent updates at 1676264467466Preparing flush snapshotting stores in 6b4351d6197a4df3ac583e92c336c18d at 1676264467466Finished memstore snapshotting testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d., syncing WAL and waiting on mvcc, flushsize=dataSize=2410, getHeapSize=5424, getOffHeapSize=0, getCellsCount=36 at 1676264467466Flushing stores of testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d. at 1676264467470 (+4 ms)Flushing 6b4351d6197a4df3ac583e92c336c18d/cf: creating writer at 1676264467470Flushing mob file 6b4351d6197a4df3ac583e92c336c18d/cf: appending metadata at 1676264467474 (+4 ms)Flushing mob file 6b4351d6197a4df3ac583e92c336c18d/cf: closing flushed file at 1676264467474Flushing 6b4351d6197a4df3ac583e92c336c18d/cf: appending metadata at 1676264467487 (+13 ms)Flushing 6b4351d6197a4df3ac583e92c336c18d/cf: closing flushed file at 1676264467487Flushing 6b4351d6197a4df3ac583e92c336c18d/cf: reopening flushed file at 1676264467503 (+16 ms)Finished flush of dataSize ~2.35 KB/2410, heapSize ~5.30 KB/5424, currentSize=0 B/0 for 6b4351d6197a4df3ac583e92c336c18d in 40ms, sequenceid=15, compaction requested=false at 1676264467506 (+3 ms)Writing region close event to WAL at 1676264467507 (+1 ms)Running coprocessor post-close hooks at 1676264467510 (+3 ms)Closed at 1676264467510 2023-02-13 05:01:07,510 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for ea5c27dda854cd4930a8099c7b035386: Waiting for close lock at 1676264467465Running coprocessor pre-close hooks at 1676264467465Disabling compacts and flushes for region at 1676264467465Disabling writes for close at 1676264467466 (+1 ms)Obtaining lock to block concurrent updates at 1676264467466Preparing flush snapshotting stores in ea5c27dda854cd4930a8099c7b035386 at 1676264467466Finished memstore snapshotting testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386., syncing WAL and waiting on mvcc, flushsize=dataSize=22307, getHeapSize=48192, getOffHeapSize=0, getCellsCount=333 at 1676264467466Flushing stores of testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386. at 1676264467468 (+2 ms)Flushing ea5c27dda854cd4930a8099c7b035386/cf: creating writer at 1676264467469 (+1 ms)Flushing mob file ea5c27dda854cd4930a8099c7b035386/cf: appending metadata at 1676264467475 (+6 ms)Flushing mob file ea5c27dda854cd4930a8099c7b035386/cf: closing flushed file at 1676264467475Flushing ea5c27dda854cd4930a8099c7b035386/cf: appending metadata at 1676264467486 (+11 ms)Flushing ea5c27dda854cd4930a8099c7b035386/cf: closing flushed file at 1676264467486Flushing ea5c27dda854cd4930a8099c7b035386/cf: reopening flushed file at 1676264467503 (+17 ms)Finished flush of dataSize ~21.78 KB/22307, heapSize ~47.06 KB/48192, currentSize=0 B/0 for ea5c27dda854cd4930a8099c7b035386 in 40ms, sequenceid=15, compaction requested=false at 1676264467507 (+4 ms)Writing region close event to WAL at 1676264467507Running coprocessor post-close hooks at 1676264467510 (+3 ms)Closed at 1676264467510 2023-02-13 05:01:07,511 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:07,511 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=276 updating hbase:meta row=ea5c27dda854cd4930a8099c7b035386, regionState=CLOSED 2023-02-13 05:01:07,511 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:07,511 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264467511"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264467511"}]},"ts":"1676264467511"} 2023-02-13 05:01:07,512 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=271 updating hbase:meta row=6b4351d6197a4df3ac583e92c336c18d, regionState=CLOSED 2023-02-13 05:01:07,512 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.","families":{"info":[{"qualifier":"regioninfo","vlen":70,"tag":[],"timestamp":"1676264467512"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1676264467512"}]},"ts":"1676264467512"} 2023-02-13 05:01:07,514 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=278, resume processing ppid=276 2023-02-13 05:01:07,514 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=278, ppid=276, state=SUCCESS; CloseRegionProcedure ea5c27dda854cd4930a8099c7b035386, server=jenkins-hbase4.apache.org,42589,1676264434015 in 250 msec 2023-02-13 05:01:07,514 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=277, resume processing ppid=271 2023-02-13 05:01:07,514 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=277, ppid=271, state=SUCCESS; CloseRegionProcedure 6b4351d6197a4df3ac583e92c336c18d, server=jenkins-hbase4.apache.org,39885,1676264434059 in 252 msec 2023-02-13 05:01:07,514 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=276, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=ea5c27dda854cd4930a8099c7b035386, UNASSIGN in 256 msec 2023-02-13 05:01:07,515 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=271, resume processing ppid=270 2023-02-13 05:01:07,515 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=271, ppid=270, state=SUCCESS; TransitRegionStateProcedure table=testCorruptedSnapshot-1676264461448, region=6b4351d6197a4df3ac583e92c336c18d, UNASSIGN in 256 msec 2023-02-13 05:01:07,516 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"testCorruptedSnapshot-1676264461448","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1676264467516"}]},"ts":"1676264467516"} 2023-02-13 05:01:07,516 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=testCorruptedSnapshot-1676264461448, state=DISABLED in hbase:meta 2023-02-13 05:01:07,519 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set testCorruptedSnapshot-1676264461448 to state=DISABLED 2023-02-13 05:01:07,521 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=270, state=SUCCESS; DisableTableProcedure table=testCorruptedSnapshot-1676264461448 in 267 msec 2023-02-13 05:01:07,738 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-02-13 05:01:08,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=270 2023-02-13 05:01:08,008 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:testCorruptedSnapshot-1676264461448, procId: 270 completed 2023-02-13 05:01:08,009 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.HMaster$5(2215): Client=jenkins//172.31.14.131 delete testCorruptedSnapshot-1676264461448 2023-02-13 05:01:08,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] procedure2.ProcedureExecutor(1029): Stored pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=testCorruptedSnapshot-1676264461448 2023-02-13 05:01:08,011 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=283, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676264461448 2023-02-13 05:01:08,011 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=283, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676264461448 2023-02-13 05:01:08,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-13 05:01:08,017 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:08,017 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8 2023-02-13 05:01:08,017 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:08,017 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:08,017 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:08,017 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:08,019 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/cf, FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/recovered.edits] 2023-02-13 05:01:08,019 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/cf, FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/recovered.edits] 2023-02-13 05:01:08,019 DEBUG [HFileArchiver-6] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/cf, FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/recovered.edits] 2023-02-13 05:01:08,020 DEBUG [HFileArchiver-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/cf, FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/recovered.edits] 2023-02-13 05:01:08,020 DEBUG [HFileArchiver-2] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/cf, FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/recovered.edits] 2023-02-13 05:01:08,020 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/cf, FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/recovered.edits] 2023-02-13 05:01:08,024 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/cf/704985202f6747a49988dad80dd7e0e8 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/cf/704985202f6747a49988dad80dd7e0e8 2023-02-13 05:01:08,025 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/cf/7022918372ea489b9310c9c7dd7ebf70 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/cf/7022918372ea489b9310c9c7dd7ebf70 2023-02-13 05:01:08,025 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/cf/df5537851dc2498f9ae0922d6a77a7b1 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/cf/df5537851dc2498f9ae0922d6a77a7b1 2023-02-13 05:01:08,025 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/cf/198c1b1681fe4d02899483305542ce83 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/cf/198c1b1681fe4d02899483305542ce83 2023-02-13 05:01:08,025 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/cf/17388fd8e87944a4a8d08c6f08a3bb35 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/cf/17388fd8e87944a4a8d08c6f08a3bb35 2023-02-13 05:01:08,026 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/cf/94e86fe7895240e9950f90ec9b3fc26b to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/cf/94e86fe7895240e9950f90ec9b3fc26b 2023-02-13 05:01:08,026 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/cf/8599a2aec0ea4917aa50e068b59c2e13 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/cf/8599a2aec0ea4917aa50e068b59c2e13 2023-02-13 05:01:08,027 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/cf/f43d7530820648cb854f1d7e7c6ee2f3 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/cf/f43d7530820648cb854f1d7e7c6ee2f3 2023-02-13 05:01:08,027 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/cf/9492210216e1415bafcf4380ca4856c1 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/cf/9492210216e1415bafcf4380ca4856c1 2023-02-13 05:01:08,027 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/cf/a20aad9ac1e34317ad796a7b1dce4b50 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/cf/a20aad9ac1e34317ad796a7b1dce4b50 2023-02-13 05:01:08,027 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/cf/decb23750b4d445e8daed2f043c924b2 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/cf/decb23750b4d445e8daed2f043c924b2 2023-02-13 05:01:08,028 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/cf/d1c935e662f84c06ad6d3e3688364edc to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/cf/d1c935e662f84c06ad6d3e3688364edc 2023-02-13 05:01:08,031 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/recovered.edits/18.seqid to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d/recovered.edits/18.seqid 2023-02-13 05:01:08,031 DEBUG [HFileArchiver-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/recovered.edits/18.seqid to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386/recovered.edits/18.seqid 2023-02-13 05:01:08,031 DEBUG [HFileArchiver-2] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/recovered.edits/18.seqid to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49/recovered.edits/18.seqid 2023-02-13 05:01:08,031 DEBUG [HFileArchiver-6] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/recovered.edits/18.seqid to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8/recovered.edits/18.seqid 2023-02-13 05:01:08,031 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/recovered.edits/18.seqid to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2/recovered.edits/18.seqid 2023-02-13 05:01:08,031 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/recovered.edits/18.seqid to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9/recovered.edits/18.seqid 2023-02-13 05:01:08,031 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/6b4351d6197a4df3ac583e92c336c18d 2023-02-13 05:01:08,032 DEBUG [HFileArchiver-1] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ea5c27dda854cd4930a8099c7b035386 2023-02-13 05:01:08,032 DEBUG [HFileArchiver-2] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/3a6771e102cb35cd838b546f1db7ee49 2023-02-13 05:01:08,032 DEBUG [HFileArchiver-6] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/ded68bdbede054e91359e396257503f8 2023-02-13 05:01:08,032 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/c6ef1845e0de3d153503ed9c1b81ead2 2023-02-13 05:01:08,032 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/.tmp/data/default/testCorruptedSnapshot-1676264461448/f11a86da37311510c40c261c7bcedac9 2023-02-13 05:01:08,032 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived testCorruptedSnapshot-1676264461448 regions 2023-02-13 05:01:08,033 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed 2023-02-13 05:01:08,033 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf] 2023-02-13 05:01:08,036 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/a87ff679a2f3e71d9181a67b7542122c202302131d035aa4667f479ab53da427dc6719e7 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/a87ff679a2f3e71d9181a67b7542122c202302131d035aa4667f479ab53da427dc6719e7 2023-02-13 05:01:08,037 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/c4ca4238a0b923820dcc509a6f75849b202302134d8391bc734f4720a4702a1800a803a1 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/c4ca4238a0b923820dcc509a6f75849b202302134d8391bc734f4720a4702a1800a803a1 2023-02-13 05:01:08,038 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/c4ca4238a0b923820dcc509a6f75849b20230213c30cbc08e63043fca92446eee8c7ca4e to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/c4ca4238a0b923820dcc509a6f75849b20230213c30cbc08e63043fca92446eee8c7ca4e 2023-02-13 05:01:08,038 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/c81e728d9d4c2f636f067f89cc14862c202302134155ae02f4584596ba6e34d042b5500a to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/c81e728d9d4c2f636f067f89cc14862c202302134155ae02f4584596ba6e34d042b5500a 2023-02-13 05:01:08,039 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/c81e728d9d4c2f636f067f89cc14862c20230213e80a32a16b364b9a812c370d393a783e to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/c81e728d9d4c2f636f067f89cc14862c20230213e80a32a16b364b9a812c370d393a783e 2023-02-13 05:01:08,040 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/d41d8cd98f00b204e9800998ecf8427e202302135713c1945093465e94004b2e52f119aa to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/d41d8cd98f00b204e9800998ecf8427e202302135713c1945093465e94004b2e52f119aa 2023-02-13 05:01:08,041 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/d41d8cd98f00b204e9800998ecf8427e20230213a1ec7568501c4cc4b8fc1f8884db77cc to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/d41d8cd98f00b204e9800998ecf8427e20230213a1ec7568501c4cc4b8fc1f8884db77cc 2023-02-13 05:01:08,042 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/e4da3b7fbbce2345d7772b0674a318d52023021314ca154df2f64d1fa8b89d8d95e3f38c to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/e4da3b7fbbce2345d7772b0674a318d52023021314ca154df2f64d1fa8b89d8d95e3f38c 2023-02-13 05:01:08,043 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/e4da3b7fbbce2345d7772b0674a318d52023021394120bb78ef749e68429ae92143d3472 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/e4da3b7fbbce2345d7772b0674a318d52023021394120bb78ef749e68429ae92143d3472 2023-02-13 05:01:08,044 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132b2a0afef79c4e7bb4a3a765d0b33947 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/eccbc87e4b5ce2fe28308fd9f2a7baf3202302132b2a0afef79c4e7bb4a3a765d0b33947 2023-02-13 05:01:08,045 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213c6986118be3c4fed80e28eba52681475 to hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/archive/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed/cf/eccbc87e4b5ce2fe28308fd9f2a7baf320230213c6986118be3c4fed80e28eba52681475 2023-02-13 05:01:08,045 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/mobdir/data/default/testCorruptedSnapshot-1676264461448/be1d479840f32cb87f1c6735788744ed 2023-02-13 05:01:08,047 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=283, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676264461448 2023-02-13 05:01:08,049 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 6 rows of testCorruptedSnapshot-1676264461448 from hbase:meta 2023-02-13 05:01:08,051 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'testCorruptedSnapshot-1676264461448' descriptor. 2023-02-13 05:01:08,051 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=283, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676264461448 2023-02-13 05:01:08,051 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'testCorruptedSnapshot-1676264461448' from region states. 2023-02-13 05:01:08,052 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264468051"}]},"ts":"9223372036854775807"} 2023-02-13 05:01:08,052 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264468051"}]},"ts":"9223372036854775807"} 2023-02-13 05:01:08,052 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264468051"}]},"ts":"9223372036854775807"} 2023-02-13 05:01:08,052 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264468051"}]},"ts":"9223372036854775807"} 2023-02-13 05:01:08,052 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264468051"}]},"ts":"9223372036854775807"} 2023-02-13 05:01:08,052 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1676264468051"}]},"ts":"9223372036854775807"} 2023-02-13 05:01:08,053 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 6 regions from META 2023-02-13 05:01:08,053 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 6b4351d6197a4df3ac583e92c336c18d, NAME => 'testCorruptedSnapshot-1676264461448,,1676264461451.6b4351d6197a4df3ac583e92c336c18d.', STARTKEY => '', ENDKEY => '1'}, {ENCODED => ded68bdbede054e91359e396257503f8, NAME => 'testCorruptedSnapshot-1676264461448,1,1676264461451.ded68bdbede054e91359e396257503f8.', STARTKEY => '1', ENDKEY => '2'}, {ENCODED => c6ef1845e0de3d153503ed9c1b81ead2, NAME => 'testCorruptedSnapshot-1676264461448,2,1676264461451.c6ef1845e0de3d153503ed9c1b81ead2.', STARTKEY => '2', ENDKEY => '3'}, {ENCODED => 3a6771e102cb35cd838b546f1db7ee49, NAME => 'testCorruptedSnapshot-1676264461448,3,1676264461451.3a6771e102cb35cd838b546f1db7ee49.', STARTKEY => '3', ENDKEY => '4'}, {ENCODED => f11a86da37311510c40c261c7bcedac9, NAME => 'testCorruptedSnapshot-1676264461448,4,1676264461451.f11a86da37311510c40c261c7bcedac9.', STARTKEY => '4', ENDKEY => '5'}, {ENCODED => ea5c27dda854cd4930a8099c7b035386, NAME => 'testCorruptedSnapshot-1676264461448,5,1676264461451.ea5c27dda854cd4930a8099c7b035386.', STARTKEY => '5', ENDKEY => ''}] 2023-02-13 05:01:08,053 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'testCorruptedSnapshot-1676264461448' as deleted. 2023-02-13 05:01:08,053 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"testCorruptedSnapshot-1676264461448","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1676264468053"}]},"ts":"9223372036854775807"} 2023-02-13 05:01:08,054 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table testCorruptedSnapshot-1676264461448 state from META 2023-02-13 05:01:08,056 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=283, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=testCorruptedSnapshot-1676264461448 2023-02-13 05:01:08,057 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=283, state=SUCCESS; DeleteTableProcedure table=testCorruptedSnapshot-1676264461448 in 47 msec 2023-02-13 05:01:08,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(1227): Checking to see if procedure is done pid=283 2023-02-13 05:01:08,263 INFO [Listener at localhost/42165] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:testCorruptedSnapshot-1676264461448, procId: 283 completed 2023-02-13 05:01:08,267 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "emptySnaptb-1676264461448" 2023-02-13 05:01:08,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(333): Deleting snapshot: emptySnaptb-1676264461448 2023-02-13 05:01:08,270 INFO [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] master.MasterRpcServices(748): Client=jenkins//172.31.14.131 delete name: "snaptb0-1676264461448" 2023-02-13 05:01:08,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45905] snapshot.SnapshotManager(333): Deleting snapshot: snaptb0-1676264461448 2023-02-13 05:01:08,288 INFO [Listener at localhost/42165] hbase.ResourceChecker(175): after: client.TestMobRestoreSnapshotFromClientSimple#testCorruptedSnapshot Thread=421 (was 428), OpenFileDescriptor=792 (was 823), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=150 (was 168), ProcessCount=170 (was 168) - ProcessCount LEAK? -, AvailableMemoryMB=3166 (was 3226) 2023-02-13 05:01:08,289 INFO [Listener at localhost/42165] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-02-13 05:01:08,289 INFO [Listener at localhost/42165] client.ConnectionImplementation(1959): Closing master protocol: MasterService 2023-02-13 05:01:08,290 DEBUG [Listener at localhost/42165] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5dd484f7 to 127.0.0.1:63943 2023-02-13 05:01:08,290 DEBUG [Listener at localhost/42165] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:01:08,290 DEBUG [Listener at localhost/42165] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-02-13 05:01:08,290 DEBUG [Listener at localhost/42165] util.JVMClusterUtil(257): Found active master hash=757396817, stopped=false 2023-02-13 05:01:08,290 INFO [Listener at localhost/42165] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,45905,1676264432900 2023-02-13 05:01:08,292 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:39885-0x1004c0d4b820002, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 05:01:08,292 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:34363-0x1004c0d4b820003, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 05:01:08,292 INFO [Listener at localhost/42165] procedure2.ProcedureExecutor(629): Stopping 2023-02-13 05:01:08,292 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 05:01:08,292 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:42589-0x1004c0d4b820001, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-02-13 05:01:08,292 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 05:01:08,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:34363-0x1004c0d4b820003, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 05:01:08,293 DEBUG [Listener at localhost/42165] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d4a255d to 127.0.0.1:63943 2023-02-13 05:01:08,293 DEBUG [Listener at localhost/42165] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:01:08,293 INFO [Listener at localhost/42165] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,42589,1676264434015' ***** 2023-02-13 05:01:08,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:39885-0x1004c0d4b820002, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 05:01:08,293 INFO [Listener at localhost/42165] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 05:01:08,293 INFO [Listener at localhost/42165] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,39885,1676264434059' ***** 2023-02-13 05:01:08,293 INFO [Listener at localhost/42165] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 05:01:08,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42589-0x1004c0d4b820001, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 05:01:08,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-02-13 05:01:08,294 INFO [RS:1;jenkins-hbase4:39885] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 05:01:08,294 INFO [Listener at localhost/42165] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,34363,1676264434089' ***** 2023-02-13 05:01:08,294 INFO [Listener at localhost/42165] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-02-13 05:01:08,294 INFO [RS:0;jenkins-hbase4:42589] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 05:01:08,294 INFO [RS:2;jenkins-hbase4:34363] regionserver.HeapMemoryManager(220): Stopping 2023-02-13 05:01:08,294 INFO [RS:0;jenkins-hbase4:42589] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 05:01:08,294 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 05:01:08,294 INFO [RS:0;jenkins-hbase4:42589] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 05:01:08,294 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 05:01:08,294 INFO [RS:0;jenkins-hbase4:42589] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:08,294 INFO [RS:1;jenkins-hbase4:39885] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 05:01:08,294 DEBUG [RS:0;jenkins-hbase4:42589] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e714ffa to 127.0.0.1:63943 2023-02-13 05:01:08,295 INFO [RS:1;jenkins-hbase4:39885] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 05:01:08,294 INFO [RS:2;jenkins-hbase4:34363] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-02-13 05:01:08,294 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-02-13 05:01:08,295 INFO [RS:2;jenkins-hbase4:34363] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-02-13 05:01:08,295 INFO [RS:1;jenkins-hbase4:39885] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:08,295 DEBUG [RS:0;jenkins-hbase4:42589] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:01:08,295 INFO [RS:2;jenkins-hbase4:34363] regionserver.HRegionServer(3304): Received CLOSE for 03df2a43656204957e15d49e0f693da9 2023-02-13 05:01:08,295 INFO [RS:0;jenkins-hbase4:42589] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,42589,1676264434015; all regions closed. 2023-02-13 05:01:08,295 DEBUG [RS:1;jenkins-hbase4:39885] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e4c05a5 to 127.0.0.1:63943 2023-02-13 05:01:08,295 DEBUG [RS:1;jenkins-hbase4:39885] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:01:08,295 INFO [RS:1;jenkins-hbase4:39885] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 05:01:08,295 INFO [RS:1;jenkins-hbase4:39885] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 05:01:08,295 INFO [RS:1;jenkins-hbase4:39885] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 05:01:08,295 INFO [RS:1;jenkins-hbase4:39885] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-02-13 05:01:08,295 INFO [RS:2;jenkins-hbase4:34363] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:08,296 DEBUG [RS:2;jenkins-hbase4:34363] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5411a7c8 to 127.0.0.1:63943 2023-02-13 05:01:08,296 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 03df2a43656204957e15d49e0f693da9, disabling compactions & flushes 2023-02-13 05:01:08,296 DEBUG [RS:2;jenkins-hbase4:34363] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:01:08,296 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9. 2023-02-13 05:01:08,296 INFO [RS:2;jenkins-hbase4:34363] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-13 05:01:08,296 INFO [RS:1;jenkins-hbase4:39885] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-02-13 05:01:08,296 DEBUG [RS:2;jenkins-hbase4:34363] regionserver.HRegionServer(1479): Online Regions={03df2a43656204957e15d49e0f693da9=hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9.} 2023-02-13 05:01:08,296 DEBUG [RS:1;jenkins-hbase4:39885] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-02-13 05:01:08,296 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9. 2023-02-13 05:01:08,296 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1603): Closing 1588230740, disabling compactions & flushes 2023-02-13 05:01:08,296 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9. after waiting 0 ms 2023-02-13 05:01:08,296 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1625): Closing region hbase:meta,,1.1588230740 2023-02-13 05:01:08,296 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9. 2023-02-13 05:01:08,296 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1646): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-02-13 05:01:08,296 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 03df2a43656204957e15d49e0f693da9 1/1 column families, dataSize=78 B heapSize=488 B 2023-02-13 05:01:08,296 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1713): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-02-13 05:01:08,296 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1723): Updates disabled for region hbase:meta,,1.1588230740 2023-02-13 05:01:08,297 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2744): Flushing 1588230740 3/3 column families, dataSize=137.71 KB heapSize=212.48 KB 2023-02-13 05:01:08,297 DEBUG [RS:2;jenkins-hbase4:34363] regionserver.HRegionServer(1505): Waiting on 03df2a43656204957e15d49e0f693da9 2023-02-13 05:01:08,297 DEBUG [RS:1;jenkins-hbase4:39885] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-02-13 05:01:08,304 DEBUG [RS:0;jenkins-hbase4:42589] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/oldWALs 2023-02-13 05:01:08,305 INFO [RS:0;jenkins-hbase4:42589] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42589%2C1676264434015:(num 1676264435552) 2023-02-13 05:01:08,305 DEBUG [RS:0;jenkins-hbase4:42589] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:01:08,305 INFO [RS:0;jenkins-hbase4:42589] regionserver.LeaseManager(133): Closed leases 2023-02-13 05:01:08,308 INFO [RS:0;jenkins-hbase4:42589] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-02-13 05:01:08,308 INFO [RS:0;jenkins-hbase4:42589] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 05:01:08,308 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 05:01:08,308 INFO [RS:0;jenkins-hbase4:42589] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 05:01:08,309 INFO [RS:0;jenkins-hbase4:42589] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 05:01:08,310 INFO [RS:0;jenkins-hbase4:42589] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42589 2023-02-13 05:01:08,324 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/namespace/03df2a43656204957e15d49e0f693da9/.tmp/info/a1a7f9c0e0c6422dbab3723fe7fe308c 2023-02-13 05:01:08,327 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/namespace/03df2a43656204957e15d49e0f693da9/.tmp/info/a1a7f9c0e0c6422dbab3723fe7fe308c as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/namespace/03df2a43656204957e15d49e0f693da9/info/a1a7f9c0e0c6422dbab3723fe7fe308c 2023-02-13 05:01:08,330 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/namespace/03df2a43656204957e15d49e0f693da9/info/a1a7f9c0e0c6422dbab3723fe7fe308c, entries=2, sequenceid=6, filesize=4.8 K 2023-02-13 05:01:08,330 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 03df2a43656204957e15d49e0f693da9 in 34ms, sequenceid=6, compaction requested=false 2023-02-13 05:01:08,341 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=130.94 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/.tmp/info/9036ba21e5184b489b82518c1492e42d 2023-02-13 05:01:08,342 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/namespace/03df2a43656204957e15d49e0f693da9/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-02-13 05:01:08,343 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9. 2023-02-13 05:01:08,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 03df2a43656204957e15d49e0f693da9: Waiting for close lock at 1676264468296Running coprocessor pre-close hooks at 1676264468296Disabling compacts and flushes for region at 1676264468296Disabling writes for close at 1676264468296Obtaining lock to block concurrent updates at 1676264468296Preparing flush snapshotting stores in 03df2a43656204957e15d49e0f693da9 at 1676264468296Finished memstore snapshotting hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9., syncing WAL and waiting on mvcc, flushsize=dataSize=78, getHeapSize=472, getOffHeapSize=0, getCellsCount=2 at 1676264468297 (+1 ms)Flushing stores of hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9. at 1676264468297Flushing 03df2a43656204957e15d49e0f693da9/info: creating writer at 1676264468297Flushing 03df2a43656204957e15d49e0f693da9/info: appending metadata at 1676264468302 (+5 ms)Flushing 03df2a43656204957e15d49e0f693da9/info: closing flushed file at 1676264468302Flushing 03df2a43656204957e15d49e0f693da9/info: reopening flushed file at 1676264468328 (+26 ms)Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 03df2a43656204957e15d49e0f693da9 in 34ms, sequenceid=6, compaction requested=false at 1676264468330 (+2 ms)Writing region close event to WAL at 1676264468333 (+3 ms)Running coprocessor post-close hooks at 1676264468342 (+9 ms)Closed at 1676264468343 (+1 ms) 2023-02-13 05:01:08,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1676264435936.03df2a43656204957e15d49e0f693da9. 2023-02-13 05:01:08,345 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9036ba21e5184b489b82518c1492e42d 2023-02-13 05:01:08,348 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 05:01:08,348 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 05:01:08,348 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-02-13 05:01:08,358 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.01 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/.tmp/rep_barrier/33f9baaa2d574d938977271875d4c93f 2023-02-13 05:01:08,361 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 33f9baaa2d574d938977271875d4c93f 2023-02-13 05:01:08,369 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.75 KB at sequenceid=307 (bloomFilter=false), to=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/.tmp/table/b8f282cc66eb45d8bfa9426bc77e5158 2023-02-13 05:01:08,372 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b8f282cc66eb45d8bfa9426bc77e5158 2023-02-13 05:01:08,372 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/.tmp/info/9036ba21e5184b489b82518c1492e42d as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/info/9036ba21e5184b489b82518c1492e42d 2023-02-13 05:01:08,375 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 9036ba21e5184b489b82518c1492e42d 2023-02-13 05:01:08,375 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/info/9036ba21e5184b489b82518c1492e42d, entries=64, sequenceid=307, filesize=11.8 K 2023-02-13 05:01:08,375 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/.tmp/rep_barrier/33f9baaa2d574d938977271875d4c93f as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/rep_barrier/33f9baaa2d574d938977271875d4c93f 2023-02-13 05:01:08,378 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 33f9baaa2d574d938977271875d4c93f 2023-02-13 05:01:08,378 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/rep_barrier/33f9baaa2d574d938977271875d4c93f, entries=18, sequenceid=307, filesize=7.0 K 2023-02-13 05:01:08,378 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/.tmp/table/b8f282cc66eb45d8bfa9426bc77e5158 as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/table/b8f282cc66eb45d8bfa9426bc77e5158 2023-02-13 05:01:08,381 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b8f282cc66eb45d8bfa9426bc77e5158 2023-02-13 05:01:08,381 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/table/b8f282cc66eb45d8bfa9426bc77e5158, entries=22, sequenceid=307, filesize=7.0 K 2023-02-13 05:01:08,381 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2947): Finished flush of dataSize ~137.71 KB/141010, heapSize ~212.43 KB/217528, currentSize=0 B/0 for 1588230740 in 85ms, sequenceid=307, compaction requested=false 2023-02-13 05:01:08,385 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/data/hbase/meta/1588230740/recovered.edits/310.seqid, newMaxSeqId=310, maxSeqId=1 2023-02-13 05:01:08,386 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-02-13 05:01:08,387 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1837): Closed hbase:meta,,1.1588230740 2023-02-13 05:01:08,387 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1557): Region close journal for 1588230740: Waiting for close lock at 1676264468296Running coprocessor pre-close hooks at 1676264468296Disabling compacts and flushes for region at 1676264468296Disabling writes for close at 1676264468296Obtaining lock to block concurrent updates at 1676264468297 (+1 ms)Preparing flush snapshotting stores in 1588230740 at 1676264468297Finished memstore snapshotting hbase:meta,,1.1588230740, syncing WAL and waiting on mvcc, flushsize=dataSize=141010, getHeapSize=217528, getOffHeapSize=0, getCellsCount=997 at 1676264468297Flushing stores of hbase:meta,,1.1588230740 at 1676264468298 (+1 ms)Flushing 1588230740/info: creating writer at 1676264468298Flushing 1588230740/info: appending metadata at 1676264468305 (+7 ms)Flushing 1588230740/info: closing flushed file at 1676264468305Flushing 1588230740/rep_barrier: creating writer at 1676264468345 (+40 ms)Flushing 1588230740/rep_barrier: appending metadata at 1676264468347 (+2 ms)Flushing 1588230740/rep_barrier: closing flushed file at 1676264468347Flushing 1588230740/table: creating writer at 1676264468361 (+14 ms)Flushing 1588230740/table: appending metadata at 1676264468363 (+2 ms)Flushing 1588230740/table: closing flushed file at 1676264468363Flushing 1588230740/info: reopening flushed file at 1676264468373 (+10 ms)Flushing 1588230740/rep_barrier: reopening flushed file at 1676264468376 (+3 ms)Flushing 1588230740/table: reopening flushed file at 1676264468379 (+3 ms)Finished flush of dataSize ~137.71 KB/141010, heapSize ~212.43 KB/217528, currentSize=0 B/0 for 1588230740 in 85ms, sequenceid=307, compaction requested=false at 1676264468382 (+3 ms)Writing region close event to WAL at 1676264468383 (+1 ms)Running coprocessor post-close hooks at 1676264468386 (+3 ms)Closed at 1676264468387 (+1 ms) 2023-02-13 05:01:08,387 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-02-13 05:01:08,401 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-13 05:01:08,401 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-13 05:01:08,403 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:34363-0x1004c0d4b820003, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:08,403 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 05:01:08,403 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:34363-0x1004c0d4b820003, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 05:01:08,403 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:42589-0x1004c0d4b820001, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:08,403 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:39885-0x1004c0d4b820002, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42589,1676264434015 2023-02-13 05:01:08,403 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:39885-0x1004c0d4b820002, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 05:01:08,403 ERROR [Listener at localhost/42165-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6c36823d rejected from java.util.concurrent.ThreadPoolExecutor@68090b63[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 05:01:08,404 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:42589-0x1004c0d4b820001, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 05:01:08,404 ERROR [Listener at localhost/42165-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@4493844c rejected from java.util.concurrent.ThreadPoolExecutor@68090b63[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 05:01:08,405 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,42589,1676264434015] 2023-02-13 05:01:08,405 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,42589,1676264434015; numProcessing=1 2023-02-13 05:01:08,406 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,42589,1676264434015 already deleted, retry=false 2023-02-13 05:01:08,406 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,42589,1676264434015 expired; onlineServers=2 2023-02-13 05:01:08,477 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-02-13 05:01:08,477 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-02-13 05:01:08,498 INFO [RS:2;jenkins-hbase4:34363] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,34363,1676264434089; all regions closed. 2023-02-13 05:01:08,498 INFO [RS:1;jenkins-hbase4:39885] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,39885,1676264434059; all regions closed. 2023-02-13 05:01:08,505 DEBUG [RS:1;jenkins-hbase4:39885] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/oldWALs 2023-02-13 05:01:08,505 INFO [RS:1;jenkins-hbase4:39885] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39885%2C1676264434059.meta:.meta(num 1676264435676) 2023-02-13 05:01:08,505 DEBUG [RS:2;jenkins-hbase4:34363] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/oldWALs 2023-02-13 05:01:08,505 INFO [RS:2;jenkins-hbase4:34363] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C34363%2C1676264434089:(num 1676264435552) 2023-02-13 05:01:08,505 DEBUG [RS:2;jenkins-hbase4:34363] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:01:08,506 INFO [RS:2;jenkins-hbase4:34363] regionserver.LeaseManager(133): Closed leases 2023-02-13 05:01:08,506 INFO [RS:2;jenkins-hbase4:34363] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 05:01:08,506 INFO [RS:2;jenkins-hbase4:34363] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-02-13 05:01:08,506 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 05:01:08,506 INFO [RS:2;jenkins-hbase4:34363] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-02-13 05:01:08,507 INFO [RS:2;jenkins-hbase4:34363] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-02-13 05:01:08,508 INFO [RS:2;jenkins-hbase4:34363] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:34363 2023-02-13 05:01:08,510 WARN [Close-WAL-Writer-0] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(641): complete file /user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/WALs/jenkins-hbase4.apache.org,39885,1676264434059/jenkins-hbase4.apache.org%2C39885%2C1676264434059.1676264435552 not finished, retry = 0 2023-02-13 05:01:08,511 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:39885-0x1004c0d4b820002, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:08,511 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:34363-0x1004c0d4b820003, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,34363,1676264434089 2023-02-13 05:01:08,511 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 05:01:08,511 ERROR [Listener at localhost/42165-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2d2e1e65 rejected from java.util.concurrent.ThreadPoolExecutor@29118f18[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 05:01:08,513 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,34363,1676264434089] 2023-02-13 05:01:08,513 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,34363,1676264434089; numProcessing=2 2023-02-13 05:01:08,514 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,34363,1676264434089 already deleted, retry=false 2023-02-13 05:01:08,514 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,34363,1676264434089 expired; onlineServers=1 2023-02-13 05:01:08,612 DEBUG [RS:1;jenkins-hbase4:39885] wal.AbstractFSWAL(932): Moved 1 WAL file(s) to /user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/oldWALs 2023-02-13 05:01:08,612 INFO [RS:1;jenkins-hbase4:39885] wal.AbstractFSWAL(935): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C39885%2C1676264434059:(num 1676264435552) 2023-02-13 05:01:08,612 DEBUG [RS:1;jenkins-hbase4:39885] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:01:08,613 INFO [RS:1;jenkins-hbase4:39885] regionserver.LeaseManager(133): Closed leases 2023-02-13 05:01:08,613 INFO [RS:1;jenkins-hbase4:39885] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-02-13 05:01:08,613 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 05:01:08,614 INFO [RS:1;jenkins-hbase4:39885] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:39885 2023-02-13 05:01:08,616 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:39885-0x1004c0d4b820002, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,39885,1676264434059 2023-02-13 05:01:08,616 ERROR [Listener at localhost/42165-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@63ce685d rejected from java.util.concurrent.ThreadPoolExecutor@67f1f0fc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 05:01:08,616 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-02-13 05:01:08,617 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,39885,1676264434059] 2023-02-13 05:01:08,617 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,39885,1676264434059; numProcessing=3 2023-02-13 05:01:08,620 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,39885,1676264434059 already deleted, retry=false 2023-02-13 05:01:08,620 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,39885,1676264434059 expired; onlineServers=0 2023-02-13 05:01:08,620 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,45905,1676264432900' ***** 2023-02-13 05:01:08,620 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-02-13 05:01:08,621 DEBUG [M:0;jenkins-hbase4:45905] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c19ff6a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-02-13 05:01:08,621 INFO [M:0;jenkins-hbase4:45905] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,45905,1676264432900 2023-02-13 05:01:08,621 INFO [M:0;jenkins-hbase4:45905] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,45905,1676264432900; all regions closed. 2023-02-13 05:01:08,621 DEBUG [M:0;jenkins-hbase4:45905] ipc.AbstractRpcClient(495): Stopping rpc client 2023-02-13 05:01:08,621 DEBUG [M:0;jenkins-hbase4:45905] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-02-13 05:01:08,621 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-02-13 05:01:08,621 DEBUG [M:0;jenkins-hbase4:45905] cleaner.HFileCleaner(317): Stopping file delete threads 2023-02-13 05:01:08,621 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676264435203] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1676264435203,5,FailOnTimeoutGroup] 2023-02-13 05:01:08,621 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676264435198] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1676264435198,5,FailOnTimeoutGroup] 2023-02-13 05:01:08,622 INFO [M:0;jenkins-hbase4:45905] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-02-13 05:01:08,622 INFO [M:0;jenkins-hbase4:45905] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-02-13 05:01:08,622 INFO [M:0;jenkins-hbase4:45905] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-02-13 05:01:08,622 DEBUG [M:0;jenkins-hbase4:45905] master.HMaster(1502): Stopping service threads 2023-02-13 05:01:08,622 INFO [M:0;jenkins-hbase4:45905] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-02-13 05:01:08,622 ERROR [M:0;jenkins-hbase4:45905] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-1,5,PEWorkerGroup] Thread[HFileArchiver-2,5,PEWorkerGroup] Thread[HFileArchiver-3,5,PEWorkerGroup] Thread[HFileArchiver-4,5,PEWorkerGroup] Thread[HFileArchiver-5,5,PEWorkerGroup] Thread[HFileArchiver-6,5,PEWorkerGroup] Thread[HFileArchiver-7,5,PEWorkerGroup] Thread[HFileArchiver-8,5,PEWorkerGroup] 2023-02-13 05:01:08,623 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-02-13 05:01:08,623 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-02-13 05:01:08,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-02-13 05:01:08,623 INFO [M:0;jenkins-hbase4:45905] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-02-13 05:01:08,623 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-02-13 05:01:08,624 DEBUG [M:0;jenkins-hbase4:45905] zookeeper.ZKUtil(398): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-02-13 05:01:08,624 WARN [M:0;jenkins-hbase4:45905] master.ActiveMasterManager(323): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-02-13 05:01:08,624 INFO [M:0;jenkins-hbase4:45905] assignment.AssignmentManager(315): Stopping assignment manager 2023-02-13 05:01:08,624 INFO [M:0;jenkins-hbase4:45905] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-02-13 05:01:08,624 DEBUG [M:0;jenkins-hbase4:45905] regionserver.HRegion(1603): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-02-13 05:01:08,624 INFO [M:0;jenkins-hbase4:45905] regionserver.HRegion(1625): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 05:01:08,624 DEBUG [M:0;jenkins-hbase4:45905] regionserver.HRegion(1646): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 05:01:08,624 DEBUG [M:0;jenkins-hbase4:45905] regionserver.HRegion(1713): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-02-13 05:01:08,624 DEBUG [M:0;jenkins-hbase4:45905] regionserver.HRegion(1723): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 05:01:08,624 INFO [M:0;jenkins-hbase4:45905] regionserver.HRegion(2744): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=845.49 KB heapSize=1.01 MB 2023-02-13 05:01:08,644 INFO [M:0;jenkins-hbase4:45905] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=845.49 KB at sequenceid=2109 (bloomFilter=true), to=hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/0ddb6d67a0374dd3b6bd28b51e124833 2023-02-13 05:01:08,646 INFO [M:0;jenkins-hbase4:45905] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0ddb6d67a0374dd3b6bd28b51e124833 2023-02-13 05:01:08,647 DEBUG [M:0;jenkins-hbase4:45905] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/0ddb6d67a0374dd3b6bd28b51e124833 as hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/0ddb6d67a0374dd3b6bd28b51e124833 2023-02-13 05:01:08,649 INFO [M:0;jenkins-hbase4:45905] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 0ddb6d67a0374dd3b6bd28b51e124833 2023-02-13 05:01:08,649 INFO [M:0;jenkins-hbase4:45905] regionserver.HStore(1080): Added hdfs://localhost:36489/user/jenkins/test-data/50b99019-c399-e21d-0f20-de46dc3d7f9a/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/0ddb6d67a0374dd3b6bd28b51e124833, entries=282, sequenceid=2109, filesize=30.9 K 2023-02-13 05:01:08,650 INFO [M:0;jenkins-hbase4:45905] regionserver.HRegion(2947): Finished flush of dataSize ~845.49 KB/865777, heapSize ~1.01 MB/1061872, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 26ms, sequenceid=2109, compaction requested=false 2023-02-13 05:01:08,651 INFO [M:0;jenkins-hbase4:45905] regionserver.HRegion(1837): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-02-13 05:01:08,651 DEBUG [M:0;jenkins-hbase4:45905] regionserver.HRegion(1557): Region close journal for 1595e783b53d99cd5eef43b6debb2682: Waiting for close lock at 1676264468624Disabling compacts and flushes for region at 1676264468624Disabling writes for close at 1676264468624Obtaining lock to block concurrent updates at 1676264468624Preparing flush snapshotting stores in 1595e783b53d99cd5eef43b6debb2682 at 1676264468624Finished memstore snapshotting master:store,,1.1595e783b53d99cd5eef43b6debb2682., syncing WAL and waiting on mvcc, flushsize=dataSize=865777, getHeapSize=1061872, getOffHeapSize=0, getCellsCount=2596 at 1676264468624Flushing stores of master:store,,1.1595e783b53d99cd5eef43b6debb2682. at 1676264468625 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: creating writer at 1676264468625Flushing 1595e783b53d99cd5eef43b6debb2682/proc: appending metadata at 1676264468630 (+5 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: closing flushed file at 1676264468631 (+1 ms)Flushing 1595e783b53d99cd5eef43b6debb2682/proc: reopening flushed file at 1676264468647 (+16 ms)Finished flush of dataSize ~845.49 KB/865777, heapSize ~1.01 MB/1061872, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 26ms, sequenceid=2109, compaction requested=false at 1676264468650 (+3 ms)Writing region close event to WAL at 1676264468651 (+1 ms)Closed at 1676264468651 2023-02-13 05:01:08,656 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-02-13 05:01:08,656 INFO [M:0;jenkins-hbase4:45905] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-02-13 05:01:08,657 INFO [M:0;jenkins-hbase4:45905] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:45905 2023-02-13 05:01:08,658 DEBUG [M:0;jenkins-hbase4:45905] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,45905,1676264432900 already deleted, retry=false 2023-02-13 05:01:08,792 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 05:01:08,793 INFO [M:0;jenkins-hbase4:45905] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,45905,1676264432900; zookeeper connection closed. 2023-02-13 05:01:08,793 ERROR [Listener at localhost/42165-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@10c366ba rejected from java.util.concurrent.ThreadPoolExecutor@24cabb2e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 28] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 05:01:08,793 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): master:45905-0x1004c0d4b820000, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 05:01:08,793 ERROR [Listener at localhost/42165-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@45d14eb6 rejected from java.util.concurrent.ThreadPoolExecutor@24cabb2e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 28] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 05:01:08,893 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:39885-0x1004c0d4b820002, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 05:01:08,893 INFO [RS:1;jenkins-hbase4:39885] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,39885,1676264434059; zookeeper connection closed. 2023-02-13 05:01:08,893 ERROR [Listener at localhost/42165-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@307584f rejected from java.util.concurrent.ThreadPoolExecutor@67f1f0fc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 05:01:08,893 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:39885-0x1004c0d4b820002, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 05:01:08,893 ERROR [Listener at localhost/42165-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7b05207d rejected from java.util.concurrent.ThreadPoolExecutor@67f1f0fc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 05:01:08,893 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@40de755f] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@40de755f 2023-02-13 05:01:08,993 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:34363-0x1004c0d4b820003, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 05:01:08,993 INFO [RS:2;jenkins-hbase4:34363] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,34363,1676264434089; zookeeper connection closed. 2023-02-13 05:01:08,993 ERROR [Listener at localhost/42165-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@b56710 rejected from java.util.concurrent.ThreadPoolExecutor@29118f18[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 05:01:08,993 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:34363-0x1004c0d4b820003, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 05:01:08,993 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@4e56d550] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@4e56d550 2023-02-13 05:01:08,993 ERROR [Listener at localhost/42165-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@78f9b1b4 rejected from java.util.concurrent.ThreadPoolExecutor@29118f18[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 05:01:09,093 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:42589-0x1004c0d4b820001, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 05:01:09,093 INFO [RS:0;jenkins-hbase4:42589] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,42589,1676264434015; zookeeper connection closed. 2023-02-13 05:01:09,093 ERROR [Listener at localhost/42165-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3c1e9374 rejected from java.util.concurrent.ThreadPoolExecutor@68090b63[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 05:01:09,093 DEBUG [Listener at localhost/42165-EventThread] zookeeper.ZKWatcher(600): regionserver:42589-0x1004c0d4b820001, quorum=127.0.0.1:63943, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-02-13 05:01:09,093 ERROR [Listener at localhost/42165-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@154d3b45 rejected from java.util.concurrent.ThreadPoolExecutor@68090b63[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-02-13 05:01:09,093 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@5c7eb574] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@5c7eb574 2023-02-13 05:01:09,094 INFO [Listener at localhost/42165] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 3 regionserver(s) complete 2023-02-13 05:01:09,094 WARN [Listener at localhost/42165] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 05:01:09,097 INFO [Listener at localhost/42165] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 05:01:09,202 WARN [BP-112057234-172.31.14.131-1676264429858 heartbeating to localhost/127.0.0.1:36489] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 05:01:09,202 WARN [BP-112057234-172.31.14.131-1676264429858 heartbeating to localhost/127.0.0.1:36489] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-112057234-172.31.14.131-1676264429858 (Datanode Uuid e3004d4a-839e-4f08-ad25-249a5adccc94) service to localhost/127.0.0.1:36489 2023-02-13 05:01:09,204 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/cluster_911aa46c-724d-9809-31e5-34f51d4314f6/dfs/data/data5/current/BP-112057234-172.31.14.131-1676264429858] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 05:01:09,204 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/cluster_911aa46c-724d-9809-31e5-34f51d4314f6/dfs/data/data6/current/BP-112057234-172.31.14.131-1676264429858] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 05:01:09,205 WARN [Listener at localhost/42165] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 05:01:09,207 INFO [Listener at localhost/42165] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 05:01:09,312 WARN [BP-112057234-172.31.14.131-1676264429858 heartbeating to localhost/127.0.0.1:36489] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 05:01:09,312 WARN [BP-112057234-172.31.14.131-1676264429858 heartbeating to localhost/127.0.0.1:36489] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-112057234-172.31.14.131-1676264429858 (Datanode Uuid 0d06ca22-fbcc-472f-b304-b58ae62322d7) service to localhost/127.0.0.1:36489 2023-02-13 05:01:09,312 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/cluster_911aa46c-724d-9809-31e5-34f51d4314f6/dfs/data/data3/current/BP-112057234-172.31.14.131-1676264429858] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 05:01:09,313 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/cluster_911aa46c-724d-9809-31e5-34f51d4314f6/dfs/data/data4/current/BP-112057234-172.31.14.131-1676264429858] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 05:01:09,313 WARN [Listener at localhost/42165] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-02-13 05:01:09,315 INFO [Listener at localhost/42165] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 05:01:09,420 WARN [BP-112057234-172.31.14.131-1676264429858 heartbeating to localhost/127.0.0.1:36489] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-02-13 05:01:09,420 WARN [BP-112057234-172.31.14.131-1676264429858 heartbeating to localhost/127.0.0.1:36489] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-112057234-172.31.14.131-1676264429858 (Datanode Uuid 9c79f770-6370-4a49-9b3f-555b35dac1f9) service to localhost/127.0.0.1:36489 2023-02-13 05:01:09,421 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/cluster_911aa46c-724d-9809-31e5-34f51d4314f6/dfs/data/data1/current/BP-112057234-172.31.14.131-1676264429858] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 05:01:09,421 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/6110eaf7-7979-860d-feb2-82e68cd75599/cluster_911aa46c-724d-9809-31e5-34f51d4314f6/dfs/data/data2/current/BP-112057234-172.31.14.131-1676264429858] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-02-13 05:01:09,454 INFO [Listener at localhost/42165] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-02-13 05:01:09,580 INFO [Listener at localhost/42165] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-02-13 05:01:09,654 INFO [Listener at localhost/42165] hbase.HBaseTestingUtility(1293): Minicluster is down